Compare commits

...

102 Commits
0.1.10 ... main

Author SHA1 Message Date
mueh e4590f3ea9 Merge branch 'main' of https://gitea.muehlberger.net/sbp-jm/sbp-flarum-search 2026-04-28 15:40:42 +02:00
mueh 83c0b0e7ec Working stable release! 2026-04-28 15:40:09 +02:00
MauBiBot 3a8559ecca Rename internal API route reference from blomstra.search to sbp.search 2026-04-28 14:46:44 +02:00
MauBiBot 082892f413 Fix forum search API endpoint to /sbp/search 2026-04-28 11:51:34 +02:00
MauBiBot 0b9586aee2 Fix forum search API endpoint to /sbp/search 2026-04-28 11:51:15 +02:00
mueh c54d916361 richitger api pfad 2026-04-28 11:49:57 +02:00
jm 09c01b3445 glaub nicht dass das böse war 2026-04-28 09:30:40 +00:00
jm a089698247 register provider evtl. böse? 2026-04-28 09:25:30 +00:00
mueh 7344fc33b7 revert to old extend.php 2026-04-28 11:06:47 +02:00
jm 2696fa4dfd wieder ohne extensionid 2026-04-28 09:00:51 +00:00
mueh a6d6230553 typo 2026-04-28 10:57:15 +02:00
mueh f537506166 Merge branch 'main' of https://gitea.muehlberger.net/sbp-jm/sbp-flarum-search 2026-04-28 10:49:51 +02:00
mueh a9bb20dfe7 extension id angegeben 2026-04-28 10:49:22 +02:00
MauBiBot 6417e8e9f4 Bind admin settings to sbp-search extension page 2026-04-28 10:31:54 +02:00
mueh 22e2796a65 admin js hatte noch alte blomstra-search drin 2026-04-28 10:27:28 +02:00
mueh 52168bb5c3 js wieder gefixt aus anderem commit 2026-04-28 10:01:50 +02:00
mueh 7f4befd73b composer fix 2026-04-28 09:37:39 +02:00
mueh 115c06e9bf Merge branch 'sbp-hardfork' 2026-04-28 09:07:16 +02:00
jm d2b8869eb4 composer update 2026-04-22 22:01:57 +00:00
MauBiBot f72acc60ef Rename blomstra/search fork to sbp/flarum-search (Phase 1: namespaces, routes, JS initializers) 2026-04-22 22:23:47 +02:00
JM fc830ea921 Add content autocomplete to search dropdown 2026-04-22 20:52:35 +02:00
MauBiBot 4538fe2ba8 Add prefix search for content and enable autocomplete 2026-04-22 20:43:33 +02:00
JM 5774879331 fuzzy search in thread 2026-04-22 20:19:09 +02:00
MauBiBot 3e2f3f9585 always use ES search on initial search page load 2026-04-22 19:46:09 +02:00
MauBiBot bf90e3279e remove frontend default sort from discussion search state 2026-04-22 19:31:30 +02:00
MauBiBot bce3258325 fix forum search route and rebuild compiled assets 2026-04-22 19:05:46 +02:00
MauBiBot 7f282cdb84 merge local search fixes after pull 2026-04-22 18:56:43 +02:00
JM 7cb4071a54 restore strong title match and keep prefix title fallback 2026-04-22 18:08:12 +02:00
JM e3dfc04192 add low-boost title autocomplete match to full search 2026-04-22 17:50:30 +02:00
MauBiBot e7e4495306 rebuild compiled JS assets for analyzer selector 2026-04-22 16:36:24 +02:00
mueh 0fa8fb92eb ai forgot to commit 2026-04-22 15:47:54 +02:00
mueh 925ecb0e7c tune title autocomplete edge_ngram bounds 2026-04-22 12:33:13 +02:00
Bart van Bragt e92307cf16 build: commit compiled JS dist files for feat/analyzer-refactor branch 2026-04-16 22:07:28 +02:00
Bart van Bragt d27d71ff72 style: apply prettier formatting to admin/index.ts 2026-04-16 11:57:59 +02:00
Bart van Bragt f7d9935fb9 feat: suppress ES instant results when search-post-bodies is enabled
When searching inside comments is ON, the autocomplete dropdown now returns
nothing from ES (only usernames show). This avoids showing title-only matches
that are inconsistent with the full comment-body search results.

Also passes filter[autocomplete]=1 to the API so the server uses the fast
title.autocomplete path instead of the full has_child query.
2026-04-16 11:57:34 +02:00
Bart van Bragt 33d4946947 feat: stem exclusion admin setting and analyzer language selector improvements
Add a stem-exclusion textarea to the admin settings page. Words entered here
(one per line) are passed as stem_exclusion to the language analyzer, preventing
them from being stemmed. Requires a full re-index to take effect.

The two language-dependent fields (stem exclusion, min search length) are
disabled when CJK is selected, since neither applies to bigram analysis.

Settings are reordered: search-scope switches now appear above the language
block so the language-dependent fields sit directly below the language selector.

Replace the lowercase ES identifier labels in the language dropdown with proper
display names (e.g. "Brazilian Portuguese", "CJK (Chinese, Japanese, Korean)",
"Sorani (Kurdish)"). Drift detection now tracks stem exclusion instead of the
removed min_gram index setting.
2026-04-16 11:52:34 +02:00
Bart van Bragt 0f58abf141 feat: add title.autocomplete instant search path sorted by latest
When filter[autocomplete]=1 is present, bypass the full has_child query
and run a single MatchQuery on title.autocomplete (edge-ngram prefix match).
Results are sorted by updated_at desc to stay consistent with full search.
2026-04-16 11:51:45 +02:00
Bart van Bragt 0adaab5d84 fix: resolve analyzer mismatch with symmetric language analyzer and title field
Replace the asymmetric ngram (index) / language (search) setup on content
with a symmetric language analyzer at both index and search time, eliminating
zero-hit cases caused by stemmed tokens falling below min_gram (e.g. Turkish).

Add a title field to discussion documents with a title.autocomplete edge-ngram
subfield (min=2, max=15) for prefix instant search, and a flarum_title_search
analyzer (lowercase only) as its search-time counterpart.

Remove flarum_analyzer_partial / partial_search_filter / search_analyzer from
the content mapping — no longer needed with symmetric analysis.

Bump INDEX_COMPAT_VERSION to v3. Also fix MatchQuery::toArray() to serialize
the analyzer property which was stored but never emitted.
2026-04-16 11:51:28 +02:00
Bart van Bragt c55db0abd6 fix: show TerminalPost in search results when field sort is active or mostRelevantPost is unavailable
When params.q is set, Flarum core suppresses TerminalPost and only shows an
excerpt from mostRelevantPost. This leaves the info section empty when the post
is null (e.g. hidden/deleted first post in production) or when results are
ordered by a field sort where an excerpt is meaningless.

Extend DiscussionListItem.infoItems to:
- Replace excerpt with TerminalPost when a field sort (latest, oldest, top) is
  active — relevance excerpts are not meaningful when sorted by date/count.
- Fall back to TerminalPost when no excerpt was added (mostRelevantPost null).
2026-04-15 21:24:55 +02:00
Bart van Bragt c5643bed31 fix: default search sort to latest and keep dropdown label in sync
- Search with no explicit sort now defaults to latest (updated_at desc)
  on both backend and frontend, matching the forum UX expectation.
- "All discussions" link from the search box routes to /?q=...&sort=latest
  so the sort dropdown shows "Latest" immediately after submission.
- extendDiscussionState injects sort=-lastPostedAt into API params when
  no sort is present, keeping direct-URL navigation (?q=anan) consistent
  with the backend default.
- Backend guard changed from phpSortField===null to empty($sorts), which
  is the correct sentinel since $sorts is the authoritative ES sort list.
2026-04-15 17:21:10 +02:00
Bart van Bragt 6bcd2becd5 fix: report actual failed item error in bulk seeding exception 2026-04-15 14:08:04 +02:00
Bart van Bragt 2706ed6be0 fix: index discussion on rename 2026-04-15 14:07:55 +02:00
Bart van Bragt 557604e426 fix: retry fill ES queries on transient connection failure 2026-04-15 14:07:43 +02:00
Bart van Bragt 08353d2473 perf: sort by date by default, skip scoring on field sorts
- Default sort changed from relevance to updated_at desc. This lets
  has_child use score_mode:none, which skips child scoring entirely
  and lets ES short-circuit early on large corpora.
- When a field sort is requested, score_mode is also none for the same
  reason. score_mode:sum is only used when sorting by relevance.
- track_total_hits:false avoids a full-index count on every query,
  allowing ES to stop once it has collected enough results.
- Strip all gambit operators (tag:foo, author:bar, is:unread, etc.)
  from the ES query string, not just is:private. Leaving them in caused
  operator:and to require the gambit tokens to appear literally in post
  content, producing zero results when gambits were combined with text.
2026-04-14 22:40:43 +02:00
Bart van Bragt 80357e6a77 fix: include --staging in build conflict suggestions 2026-04-14 16:46:06 +02:00
Bart van Bragt d8eebb183a fix: prevent ES CPU saturation from has_child scoring all posts
Without minimum_should_match=1 on the inner post bool query, Elasticsearch
defaults MSM to 0 whenever a filter clause is present. This caused has_child
to score every non-hidden post on every search request, saturating CPU on
shared ES nodes with large corpora (3.7M+ posts).

- Add BoolQuery subclass with create() override and minimumShouldMatch()
  (spatie/elasticsearch-query-builder 1.x uses `new self()` in create(),
  so subclassing requires overriding it)
- Set minimumShouldMatch(1) on the inner post query after adding the
  is_hidden filter, so only genuinely matching posts are scored
- Remove the operator('or') clause from buildShouldClauses() — with
  Turkish min_ngram=2, 'or' generates 2-gram tokens that match nearly
  every post, causing near-total index scans
- Add ES client timeouts (connect: 2s, query: 10s) to prevent Apache
  mod_php worker saturation when ES is slow or unreachable
2026-04-14 16:07:50 +02:00
Bart van Bragt c54f37349f fix: use Support\Collection for cachedGlobalPermission — pluck() returns plain Collection, not Eloquent 2026-04-14 11:27:02 +02:00
Bart van Bragt ca8c248870 fix: use Eloquent Collection in job — loadMissing does not exist on Support\Collection 2026-04-14 09:59:02 +02:00
Bart van Bragt bf37e01eb0 fix: read forum attribute lazily inside extend callback 2026-04-14 09:33:17 +02:00
Bart van Bragt 4709b9d1db fix: is_hidden boolean coercion and forum settings access 2026-04-14 08:59:19 +02:00
flarum-bot bf19eed20f Bundled output for commit 2fc66b9bf7
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2026-04-13 20:22:53 +00:00
Bart van Bragt 2fc66b9bf7 perf: eager-load relationships in UpdateSearchJob and cache permissions in DiscussionSeeder
- Add relationships() to Seeder base class; DiscussionSeeder implements it to
  return tags/recipientUsers/recipientGroups for eager loading
- UpdateSearchJob calls loadMissing(relationships()) before the map loop,
  eliminating N×R lazy-load queries per batch
- Remove refresh:true from bulk call — synchronous ES refresh was the primary
  throughput bottleneck during indexing
- Cache viewForum permissions in DiscussionSeeder per job instance, replacing
  N×2 Permission queries per document with a single cached query
- Increase seeder batch size from 1000 to 2500
2026-04-13 22:18:14 +02:00
Bart van Bragt 439c74f0a3 fix: store index compat version in ES mapping _meta for correct rollback behaviour
- Write INDEX_COMPAT_VERSION into mapping _meta.index_compat_version on every
  build/promote so the version travels with the index rather than being a
  separate settings value
- saveIndexedConfig now reads compat version back from _meta alongside the
  existing analyzer/min_gram sync, so rollback automatically restores the
  correct value (or null for pre-feature indexes, triggering the reindex warning)
- Replace InvalidArgumentException in buildIndexSettings with $this->error()
  fallback to avoid stack traces in the console
- Fix stale comment in ViewsSearchJob referencing removed CLI flags
2026-04-10 13:46:08 +02:00
Bart van Bragt f486292f15 feat: make ngram min_gram configurable via admin setting
Adds a 'Minimum search query length' setting (1–4, default 3). The admin
shows a warning when the analyzer or min-search-length have changed since
the last index build.
2026-04-10 13:19:22 +02:00
Bart van Bragt 4412583cd9 chore: fix prettier formatting and npm audit vulnerabilities
Reformat admin/index.ts to satisfy prettier checks. Update package-lock.json
via npm audit fix to resolve 17 vulnerabilities in transitive dev dependencies
(webpack, babel, lodash, ws, semver, etc.).
2026-04-10 11:47:38 +02:00
Bart van Bragt 81e7b6502d feat: Show admin dashboard notice when search index needs rebuilding
After a mapping change (e.g. the parent-child restructure), existing installs
need to rebuild their index. A DashboardWidget appears on the admin page
warning the admin and showing the command to run. Compatibility is tracked
via the blomstra-search.index-compatible setting.
2026-04-10 11:10:53 +02:00
Bart van Bragt 3039b65946 refactor: Simplify index command with better naming/options
The flag-based interface (--rebuild etc) became rather convoluted with the changes. Replace this with commands with more straight forward naming.

Also added an explicit 'rollback' action and updated the README.md
2026-04-10 09:35:49 +02:00
Bart van Bragt e2e2d1428f feat: restructure index as parent-child documents with blue-green rebuilds
- Switch to ES parent/child join_field: discussion docs hold all metadata,
  post docs hold content only. Routing ensures parent and child land on the
  same shard. Removes the old flat-document approach where post data was
  duplicated onto every comment document.

- Add HasChildQuery with inner_hits so the best-matching post ID is
  surfaced as mostRelevantPost without a second DB query.

- Blue-green index rebuilds: --recreate writes into a timestamped pending
  index, --swap atomically promotes it via alias. Interrupted builds are
  resumable with --recreate --continue. --swap requires confirmation before
  proceeding.

- Index hidden posts; non-moderators are filtered at query time via
  is_hidden on the has_child clause. Core\Hidden and Core\Restored trigger
  re-indexing so moderators with post.hide can search hidden posts.
2026-04-09 23:30:57 +02:00
Bart van Bragt 80d66c98c4 feat: index and sync view_count for fof/discussion-views sort support
- Rename SavingJob to UpdateSearchJob
- Add ViewsSearchJob for lightweight partial view_count updates
- Add probabilistic sync throttle in DiscussionSeeder::viewingOn()
  to avoid an ES update on every page view
- Add view_count to mapping, DiscussionSeeder::toDocument(), and
  translateSort
- Log a warning and skip instead of 500ing on unknown sort fields
2026-04-09 12:05:28 +02:00
Bart van Bragt ff2a252e99 fix: editing a post does not update the search index
CommentSeeder only listened to Post\Event\Posted, so post edits
left the Elasticsearch document stale.
2026-04-09 11:17:17 +02:00
Bart van Bragt 5929a191f6 fix: --seed-missing stops early when a full range is already indexed
When all documents in a range were already in the index, the empty
collection caused continueAt to be set to null, aborting before
scanning lower ID ranges. Now advances past the range bottom instead.
2026-04-09 11:17:17 +02:00
Bart van Bragt 6699c8cb58 fix: ngram min_gram=1 indexes tokens that can never be matched
Flarum's search UI enforces a minimum query length of 3 characters
(MIN_SEARCH_LEN in Search.tsx), so 1- and 2-character ngram tokens
are indexed but can never be queried. Changed min_gram from 1 to 3
and updated max_ngram_diff accordingly.
2026-04-09 11:04:27 +02:00
Bart van Bragt 369ee504f3 fix: remove unused content_partial field from seeders
content_partial was introduced alongside a partialMatch() query that
targeted it. That query was later switched to use the content field
instead, but content_partial was never removed from the seeders,
causing every document to store and index its content twice.
2026-04-09 08:39:58 +02:00
Bart van Bragt eeb091273d fix: tag filter produces empty results when browsing a tag
When navigating to /t/{tag}?q=query the tag filter from the request
was silently ignored. Added a TermsQuery on the tags field using the
tag IDs resolved from the provided slugs.
2026-04-09 08:20:42 +02:00
Daniël Klabbers d252a17586 fix: not changing recipients visibility when recipients change 2024-06-13 19:27:14 +02:00
Daniël Klabbers a9dc8bfb16 fix: byobu discussions open up while using this extension 2024-06-12 11:34:18 +02:00
StyleCI Bot 82e6ec786f
Apply fixes from StyleCI 2023-10-18 13:57:41 +00:00
flarum-bot f981c27e00 Bundled output for commit f6503f5e6a
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2023-01-30 22:17:40 +00:00
Daniël Klabbers f6503f5e6a fix: #10, private forums have no discussions key, throws error 2023-01-30 23:17:01 +01:00
flarum-bot ad92d8856b Bundled output for commit 8b519619e5
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-24 09:39:12 +00:00
StyleCI Bot 8b519619e5
Apply fixes from StyleCI 2022-11-24 09:38:21 +00:00
Daniël Klabbers 83afe95d61 chore: fix search and seeding 2022-11-24 10:37:57 +01:00
Daniël Klabbers 846cc348ff Merge branch 'dk/search-improvements' into main 2022-11-24 10:29:18 +01:00
Daniël Klabbers 508bb42ac9 chore: clean up buildcommand 2022-11-23 12:46:04 +01:00
StyleCI Bot db74da3e37
Apply fixes from StyleCI 2022-11-23 11:29:48 +00:00
Daniël Klabbers a02d35394b feat: store tags, rawId, support indexing missing objects 2022-11-23 12:09:10 +01:00
flarum-bot b60563f2ec Bundled output for commit 703cbbdd48
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-21 15:33:18 +00:00
David Wheatley 703cbbdd48
fix: incorrect JS package name 2022-11-21 15:32:22 +00:00
flarum-bot 769b6cd278 Bundled output for commit f500d7f956
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-21 12:38:34 +00:00
Ian Morland f500d7f956 fix: revert back to not using the elastic endpoint for filtering, at least until we support tags in the search database 2022-11-21 12:37:52 +00:00
flarum-bot 7257d5fb34 Bundled output for commit 09fb4e8b7b
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-18 09:39:40 +00:00
Daniël Klabbers 09fb4e8b7b
Merge pull request #9 from blomstra/im/fix-search-override
fix: override search not fired correctly
2022-11-18 10:38:47 +01:00
Ian Morland 21170fbe1b Merge remote-tracking branch 'origin/main' into im/fix-search-override 2022-11-18 09:37:37 +00:00
StyleCI Bot 1cd3dac177
Apply fixes from StyleCI 2022-11-18 09:34:30 +00:00
Ian Morland 5815144d3a Revert "Apply fixes from StyleCI"
This reverts commit a26f88a9b6.
2022-11-18 09:34:16 +00:00
StyleCI Bot f8aa3dcf13
Apply fixes from StyleCI 2022-11-18 09:33:13 +00:00
StyleCI Bot a26f88a9b6
Apply fixes from StyleCI 2022-11-18 09:29:46 +00:00
Ian Morland 23dd403417 fix: override endpoint not used correctly 2022-11-18 09:29:19 +00:00
Ian Morland ed718e1458 chore: fix incorect command 2022-11-18 09:28:52 +00:00
flarum-bot 4240269b7f Bundled output for commit c186535bbb
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-18 08:31:43 +00:00
Ian Morland c186535bbb chore: use npm 2022-11-18 08:30:44 +00:00
Ian Morland fa90def995 chore: update frontend workflow 2022-11-18 08:29:19 +00:00
Ian Morland 4d2c5cb22f chore: bump deps, npm audit fix 2022-11-18 08:26:26 +00:00
Daniël Klabbers 9401b677a3 change weighing for exact words match 2022-11-10 12:57:46 +01:00
Daniël Klabbers 4bc89949ec complete testing 2022-11-10 12:54:26 +01:00
Daniël Klabbers 29ba864849 settings configurations 2022-11-09 18:06:48 +01:00
flarum-bot aabba9c2d0 Bundled output for commit 886e78d6fb
Includes transpiled JS/TS.

[skip ci]
2022-05-03 08:15:09 +00:00
Ian Morland 886e78d6fb Format js 2022-05-03 09:13:50 +01:00
Ian Morland af85834a5a Webpack 5 2022-05-03 09:13:10 +01:00
Daniël Klabbers e26d90d7b6 fix ambiguous 2022-03-04 12:37:43 +01:00
Daniël Klabbers cb3d4f5225 fix typo 2022-02-28 20:33:01 +01:00
Daniël Klabbers c6fd942bf1 prevent leaking deleted discussions 2022-02-28 20:32:27 +01:00
45 changed files with 4696 additions and 8941 deletions

View File

@ -1,17 +1,18 @@
name: Build JavaScript assets
name: Search JS
on:
push:
branches:
- main
on: [workflow_dispatch, push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@master
- uses: flarum/action-build@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run:
uses: flarum/framework/.github/workflows/REUSABLE_frontend.yml@main
with:
enable_bundlewatch: false
enable_prettier: true
enable_typescript: false
frontend_directory: ./js
backend_directory: .
js_package_manager: npm
main_git_branch: main
secrets:
bundlewatch_github_token: ${{ secrets.BUNDLEWATCH_GITHUB_TOKEN }}

1
.gitignore vendored
View File

@ -2,3 +2,4 @@ js/node_modules
js/dist
vendor/
composer.lock
analysis/*

166
README.md
View File

@ -5,47 +5,177 @@ fulltext search with one that is completely relying on the proven elasticsearch
## Features
- Sync discussions to elastic search using your queue, unobtrusively for the user.
- Reduces search loading times to well below 400ms (local tests with 50.000 discussion **260ms**)
- Sync discussions and posts to Elasticsearch using your queue, unobtrusively for the user.
- Reduces search loading times to well below 400ms (local tests with 50,000 discussions: **260ms**)
- Uses Flarum's group permissions and tags system.
- Compatible with Friends of Flarum Byōbu.
## Requirements
- Elasticsearch 7.x or OpenSearch 1.x+
- A non-sync queue driver with a running worker (`php flarum queue:work`) is strongly recommended for production. The extension works with the default sync driver, but index jobs run inline which adds latency to user-facing changes like posting.
## Installation
Use composer:
```bash
composer require sbp/flarum-search
```
## Notes
This fork is now maintained as `sbp/flarum-search`.
Intentional compatibility leftovers:
- settings keys remain `blomstra-search.*`
- some translation keys remain `blomstra-search.*`
- CLI command names may still remain `blomstra:*` until a later migration
Search behavior:
- title autocomplete enabled
- fuzzy title search enabled
- exact + fuzzy content search on full search page
- no content autocomplete in dropdown (performance decision)
``
Repository can be found here: https://gitea.muehlberger.net/sbp-jm/sbp-flarum-search
Enable the extension in the admin area and configure the Elasticsearch endpoint and index name in the extension settings.
## Setting up the index
### First install
Run the build command once. It creates a timestamped concrete index, immediately aliases your configured index name to it, and begins queuing documents. Search is available and improves as the queue processes:
```bash
composer require blomstra/search:*
php flarum blomstra:search:index build
php flarum queue:work
```
Enable the extension inside the admin area and configure the settings.
### Subsequent rebuilds (zero-downtime)
### Set up
When you need to rebuild the full index (e.g. after a mapping change):
Enable the extension in your admin area. Now to seed your existing discussions use the following command:
```bash
# Simple rebuild — promotes automatically once all jobs are queued
php flarum blomstra:search:index build
```
php flarum blomstra:search:build
# Or keep a backup of the old index in case you need to roll back
php flarum blomstra:search:index build --keep-backup
```
All mutations to discussions are automatically added and removed from the elasticsearch index.
After the queue drains, fill any gaps from content posted during the build:
### FAQ
```bash
php flarum blomstra:search:index fill
```
*I have another question.*
Reach out to us via https://helpdesk.blomstra.net. We will get back to you as soon as we can. If you have a running subscription please mention when you started your plan and/or which plan you are on. Always add sufficient information when reporting errors. We prefer errors being reported here, but understand that sometimes you can't.
If you kept a backup and want to roll back:
```bash
php flarum blomstra:search:index rollback
```
Once satisfied with the new index, drop the backup:
```bash
php flarum blomstra:search:index discard --backup
```
### Blue-green rebuild (manual promotion)
Use `--staging` to keep the old index live until you explicitly promote:
```bash
# 1. Build into a staging index — live index is untouched
php flarum blomstra:search:index build --staging
# 2. Drain the queue
php flarum queue:work --stop-when-empty
# 3. Promote the staging index to live
php flarum blomstra:search:index promote
# Or keep the old index as a backup:
php flarum blomstra:search:index promote --keep-backup
```
### Resuming or cancelling an interrupted build
If a build is interrupted, re-run it with the appropriate flag:
```bash
# Resume each seeder from where it left off
php flarum blomstra:search:index build --resume
# Drop the staging index and start completely fresh
php flarum blomstra:search:index build --fresh
# Cancel the build without starting a new one
php flarum blomstra:search:index discard --pending
```
### Filling gaps in an existing index
If documents are missing from the live index (e.g. due to queue failures):
```bash
php flarum blomstra:search:index fill
```
### Updating the mapping only
To push a mapping change to the live index without rebuilding:
```bash
php flarum blomstra:search:index mapping
```
## Command reference
| Command | Description |
|---|---|
| `build` | Rebuild the index and promote automatically once all jobs are queued. On first install, aliases immediately so search is live during seeding. |
| `build --keep-backup` | Rebuild and promote, retaining the old index as a backup for rollback. |
| `build --staging` | Build into a staging index without promoting — use `promote` when ready (blue-green workflow). |
| `build --resume` | Resume an interrupted build from where each seeder left off. |
| `build --fresh` | Drop the staging index and start completely fresh. |
| `promote` | Atomically swap the alias to the staging index. Prompts for confirmation (blue-green workflow). |
| `promote --keep-backup` | Promote and retain the replaced live index as a backup for rollback. |
| `rollback` | Restore the backup index to live. Deletes the index that was live. |
| `discard --pending` | Drop the staging index without promoting (cancels an in-progress build). |
| `discard --backup` | Drop the backup index (cleanup after `--keep-backup`). |
| `mapping` | Push updated mapping to the live index without rebuilding or reseeding. |
| `fill` | Seed only documents missing from the live index. |
| `build --only=discussions` | Seed only the specified document type (`discussions` or `posts`). |
| `build --throttle=N` | Wait N seconds between batches (reduces queue pressure). |
| `build --max-id=N` | Limit seeding to documents with ID ≤ N. |
| `promote --i-am-sure` | Skip the promotion confirmation prompt (for scripts and CI). |
## FAQ
## Queue configuration
*"Can I dispatch indexing jobs to a specific queue?"*
*Can I dispatch the sync jobs to another queue?*
Yes:
```php
\Blomstra\Search\Observe\Job::$onQueue = 'sync';
\Blomstra\Search\Jobs\Job::$onQueue = 'search';
```
*"I have a different question"*
Reach out ot us via https://support.on-floxum.com/t/ext-search . If you have an active subscription, please mention what plan you are on.
---
- Blomstra provides managed Flarum hosting.
- https://blomstra.net
- https://blomstra.community/t/ext-search
- Floxum provides managed Flarum hosting.
- https://floxum.com
- https://support.on-floxum.com/t/ext-search
Icon made by [Freepik](https://www.freepik.com) from [Flaticon](https://www.flaticon.com/).

View File

@ -1,5 +1,5 @@
{
"name": "blomstra/search",
"name": "sbp/flarum-search",
"description": "Replaces Flarum search with one powered by an elastic search server.",
"keywords": [
"extension",
@ -11,21 +11,25 @@
"elasticsearch"
],
"support": {
"email": "helpdesk@blomstra.net",
"email": "mueh+git@sbp.de",
"forum": "https://blomstra.community/t/ext-search"
},
"type": "flarum-extension",
"license": "MIT",
"authors": [
{
"name": "sbp",
"email": "info@sbp.de"
},
{
"name": "Team Blomstra",
"email": "development@blomstra.net",
"role": "Developer"
"role": "Original Developer"
}
],
"require": {
"php": ">= 8.0",
"flarum/core": "^1.0.0",
"flarum/core": "^1.2.0",
"elasticsearch/elasticsearch": "7.*",
"spatie/elasticsearch-query-builder": "^1.3.0"
},
@ -43,7 +47,7 @@
},
"autoload": {
"psr-4": {
"Blomstra\\Search\\": "src/"
"Sbp\\Search\\": "src/"
}
}
}

View File

@ -1,23 +1,41 @@
<?php
namespace Blomstra\Search;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search;
use Flarum\Extend as Flarum;
return [
(new Flarum\ServiceProvider)->register(Provider::class),
(new Flarum\ServiceProvider())->register(Provider::class),
(new Flarum\Frontend('forum'))
->js(__DIR__ . '/js/dist/forum.js'),
->js(__DIR__.'/js/dist/forum.js'),
(new Flarum\Frontend('admin'))
->js(__DIR__ . '/js/dist/admin.js'),
->js(__DIR__.'/js/dist/admin.js')
->css(__DIR__.'/resources/less/admin.less'),
(new Flarum\Locales(__DIR__ . '/resources/locale')),
new Flarum\Locales(__DIR__.'/resources/locale'),
(new Flarum\Routes('api'))
->get('/blomstra/search/{type}', 'blomstra.search', Api\Controllers\SearchController::class)
->put('/blomstra/search/index', 'blomstra.search.index', Api\Controllers\IndexController::class),
->get('/sbp/search/{type}', 'sbp.search', Api\Controllers\SearchController::class)
->put('/sbp/search/index', 'sbp.search.index', Api\Controllers\IndexController::class),
(new Flarum\Console())
->command(Commands\BuildCommand::class),
(new Flarum\Console)
->command(Commands\BuildCommand::class)
(new Flarum\Settings())
->default('blomstra-search.search-discussion-subjects', true)
->default('blomstra-search.search-post-bodies', true)
->default('blomstra-search.min-search-length', Commands\BuildCommand::DEFAULT_MIN_SEARCH_LENGTH)
->default('blomstra-search.stem-exclusion', '')
->serializeToForum('blomstraSearchMinLength', 'blomstra-search.min-search-length', 'intval')
->serializeToForum('blomstraSearchPostBodies', 'blomstra-search.search-post-bodies', fn ($v) => (bool) $v, true),
];

2
js/dist/admin.js vendored
View File

@ -1,2 +1,2 @@
module.exports=function(e){var t={};function a(r){if(t[r])return t[r].exports;var n=t[r]={i:r,l:!1,exports:{}};return e[r].call(n.exports,n,n.exports,a),n.l=!0,n.exports}return a.m=e,a.c=t,a.d=function(e,t,r){a.o(e,t)||Object.defineProperty(e,t,{enumerable:!0,get:r})},a.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},a.t=function(e,t){if(1&t&&(e=a(e)),8&t)return e;if(4&t&&"object"==typeof e&&e&&e.__esModule)return e;var r=Object.create(null);if(a.r(r),Object.defineProperty(r,"default",{enumerable:!0,value:e}),2&t&&"string"!=typeof e)for(var n in e)a.d(r,n,function(t){return e[t]}.bind(null,n));return r},a.n=function(e){var t=e&&e.__esModule?function(){return e.default}:function(){return e};return a.d(t,"a",t),t},a.o=function(e,t){return Object.prototype.hasOwnProperty.call(e,t)},a.p="",a(a.s=11)}({1:function(e,t){e.exports=flarum.core.compat["admin/app"]},11:function(e,t,a){"use strict";a.r(t);var r=a(1),n=a.n(r);n.a.initializers.add("blomstra-search",(function(){var e=new Map;["arabic","armenian","basque","bengali","brazilian","bulgarian","catalan","cjk","czech","danish","dutch","english","estonian","finnish","french","galician","german","greek","hindi","hungarian","indonesian","irish","italian","latvian","lithuanian","norwegian","persian","portuguese","romanian","russian","sorani","spanish","swedish","turkish","thai"].forEach((function(t){e.set(t,t)})),n.a.extensionData.for("blomstra-search").registerSetting({setting:"blomstra-search.elastic-endpoint",label:n.a.translator.trans("blomstra-search.admin.elastic-endpoint"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-username",label:n.a.translator.trans("blomstra-search.admin.elastic-username"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-password",label:n.a.translator.trans("blomstra-search.admin.elastic-password"),type:"password"}).registerSetting({setting:"blomstra-search.elastic-index",label:n.a.translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.analyzer-language",label:n.a.translator.trans("blomstra-search.admin.analyzer.label"),help:n.a.translator.trans("blomstra-search.admin.analyzer.help"),type:"select",options:Object.fromEntries(e.entries()),default:"english"}).registerSetting({setting:"blomstra-search.elastic-index",label:n.a.translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"})}))}});
(()=>{var a={n:t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return a.d(e,{a:e}),e},d:(t,e)=>{for(var n in e)a.o(e,n)&&!a.o(t,n)&&Object.defineProperty(t,n,{enumerable:!0,get:e[n]})},o:(a,t)=>Object.prototype.hasOwnProperty.call(a,t)};(()=>{"use strict";function t(a,e){return t=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(a,t){return a.__proto__=t,a},t(a,e)}const e=flarum.core.compat["admin/app"];var n=a.n(e);const r=flarum.core.compat["common/extend"],s=flarum.core.compat["admin/components/DashboardPage"];var i=a.n(s);const l=flarum.core.compat["admin/components/DashboardWidget"];var o=a.n(l);const c=flarum.core.compat["common/components/Alert"];var h=a.n(c),d=function(a){var e,r;function s(){return a.apply(this,arguments)||this}r=a,(e=s).prototype=Object.create(r.prototype),e.prototype.constructor=e,t(e,r);var i=s.prototype;return i.className=function(){return"ReindexWarningWidget"},i.content=function(){return m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle",title:n().translator.trans("blomstra-search.admin.reindex-required.title")},n().translator.trans("blomstra-search.admin.reindex-required.detail"))},s}(o());n().initializers.add("sbp-search",function(){var a=n().data.settings["blomstra-search.active-index"],t=n().data.settings["blomstra-search.index-compatible"];a&&"v3"!==t&&(0,r.extend)(i().prototype,"availableWidgets",function(a){a.add("blomstra-search-reindex",m(d),110)}),n().extensionData.for("sbp-search").registerSetting(function(){var t=n().data.settings["blomstra-search.indexed-analyzer"];if(!a||!t)return null;var e=this.setting("blomstra-search.analyzer-language")()||"english",r=this.setting("blomstra-search.stem-exclusion")()||"",s=n().data.settings["blomstra-search.indexed-stem-exclusion"]||"";return e===t&&r===s?null:m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle"},n().translator.trans("blomstra-search.admin.index-settings-changed"))}).registerSetting({setting:"blomstra-search.elastic-endpoint",label:n().translator.trans("blomstra-search.admin.elastic-endpoint"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-username",label:n().translator.trans("blomstra-search.admin.elastic-username"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-password",label:n().translator.trans("blomstra-search.admin.elastic-password"),type:"password"}).registerSetting({setting:"blomstra-search.elastic-index",label:n().translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.search-discussion-subjects",label:n().translator.trans("blomstra-search.admin.search-discussion-subjects"),type:"switch"}).registerSetting({setting:"blomstra-search.search-post-bodies",label:n().translator.trans("blomstra-search.admin.search-post-bodies"),type:"switch"}).registerSetting({setting:"blomstra-search.analyzer-language",label:n().translator.trans("blomstra-search.admin.analyzer.label"),help:n().translator.trans("blomstra-search.admin.analyzer.help"),type:"select",options:{arabic:"Arabic",armenian:"Armenian",basque:"Basque",bengali:"Bengali",brazilian:"Brazilian Portuguese",bulgarian:"Bulgarian",catalan:"Catalan",cjk:"CJK (Chinese, Japanese, Korean)",czech:"Czech",danish:"Danish",dutch:"Dutch",english:"English",estonian:"Estonian",finnish:"Finnish",french:"French",galician:"Galician",german:"German",greek:"Greek",hindi:"Hindi",hungarian:"Hungarian",indonesian:"Indonesian",irish:"Irish",italian:"Italian",latvian:"Latvian",lithuanian:"Lithuanian",norwegian:"Norwegian",persian:"Persian",portuguese:"Portuguese",romanian:"Romanian",russian:"Russian",sorani:"Sorani (Kurdish)",spanish:"Spanish",standard:"Standard (generic)",swedish:"Swedish",turkish:"Turkish",thai:"Thai"},default:"english"}).registerSetting(function(){var a="cjk"===(this.setting("blomstra-search.analyzer-language")()||"english");return this.buildSettingComponent({setting:"blomstra-search.stem-exclusion",type:"textarea",label:n().translator.trans("blomstra-search.admin.settings.stem-exclusion.label"),help:n().translator.trans("blomstra-search.admin.settings.stem-exclusion.help"),disabled:a})}).registerSetting(function(){var a="cjk"===(this.setting("blomstra-search.analyzer-language")()||"english");return this.buildSettingComponent({setting:"blomstra-search.min-search-length",label:n().translator.trans("blomstra-search.admin.min-search-length.label"),help:n().translator.trans("blomstra-search.admin.min-search-length.help"),type:"select",options:{1:"1",2:"2",3:"3",4:"4"},default:n().data.settings["blomstra-search.min-search-length"],disabled:a})})})})(),module.exports={}})();
//# sourceMappingURL=admin.js.map

File diff suppressed because one or more lines are too long

3
js/dist/forum.js vendored

File diff suppressed because one or more lines are too long

1
js/dist/forum.js.LICENSE.txt vendored Normal file
View File

@ -0,0 +1 @@
/*! regenerator-runtime -- Copyright (c) 2014-present, Facebook, Inc. -- license (MIT): https://github.com/babel/babel/blob/main/packages/babel-helpers/LICENSE */

File diff suppressed because one or more lines are too long

10811
js/package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@ -1,16 +1,14 @@
{
"name": "@blomstra/realtime",
"name": "@blomstra/search",
"private": true,
"version": "0.0.0",
"prettier": "@flarum/prettier-config",
"dependencies": {
"@flarum/prettier-config": "^1.0.0",
"build": "^0.1.4",
"flarum-tsconfig": "^1.0.2",
"flarum-webpack-config": "^1.0.0",
"pusher-js": "^7.0.3",
"webpack": "^4.46.0",
"webpack-cli": "^4.9.1"
"flarum-webpack-config": "^2.0.0",
"webpack": "^5.75.0",
"webpack-cli": "^5.0.0"
},
"scripts": {
"dev": "webpack --mode development --watch",
@ -19,6 +17,6 @@
"format-check": "prettier --check src"
},
"devDependencies": {
"prettier": "^2.4.1"
"prettier": "^2.7.1"
}
}

View File

@ -1,50 +1,157 @@
import app from 'flarum/admin/app';
import { extend } from 'flarum/common/extend';
import DashboardPage from 'flarum/admin/components/DashboardPage';
import DashboardWidget from 'flarum/admin/components/DashboardWidget';
import Alert from 'flarum/common/components/Alert';
app.initializers.add('blomstra-search', () => {
const languages = new Map;
['arabic', 'armenian', 'basque', 'bengali', 'brazilian', 'bulgarian', 'catalan', 'cjk', 'czech',
'danish', 'dutch', 'english', 'estonian', 'finnish', 'french', 'galician', 'german', 'greek',
'hindi', 'hungarian', 'indonesian', 'irish', 'italian', 'latvian', 'lithuanian', 'norwegian',
'persian', 'portuguese', 'romanian', 'russian', 'sorani', 'spanish', 'swedish', 'turkish', 'thai']
.forEach(language => {
languages.set(language, language);
const REQUIRED_INDEX_COMPAT = 'v3';
class ReindexWarningWidget extends DashboardWidget {
className() {
return 'ReindexWarningWidget';
}
content() {
return m(
Alert,
{
type: 'warning',
dismissible: false,
icon: 'fas fa-exclamation-triangle',
title: app.translator.trans('blomstra-search.admin.reindex-required.title'),
},
app.translator.trans('blomstra-search.admin.reindex-required.detail')
);
}
}
app.initializers.add('sbp-search', () => {
const activeIndex = app.data.settings['blomstra-search.active-index'];
const compatVersion = app.data.settings['blomstra-search.index-compatible'];
if (activeIndex && compatVersion !== REQUIRED_INDEX_COMPAT) {
extend(DashboardPage.prototype, 'availableWidgets', function (items) {
items.add('blomstra-search-reindex', m(ReindexWarningWidget), 110);
});
}
const languages: Record<string, string> = {
arabic: 'Arabic',
armenian: 'Armenian',
basque: 'Basque',
bengali: 'Bengali',
brazilian: 'Brazilian Portuguese',
bulgarian: 'Bulgarian',
catalan: 'Catalan',
cjk: 'CJK (Chinese, Japanese, Korean)',
czech: 'Czech',
danish: 'Danish',
dutch: 'Dutch',
english: 'English',
estonian: 'Estonian',
finnish: 'Finnish',
french: 'French',
galician: 'Galician',
german: 'German',
greek: 'Greek',
hindi: 'Hindi',
hungarian: 'Hungarian',
indonesian: 'Indonesian',
irish: 'Irish',
italian: 'Italian',
latvian: 'Latvian',
lithuanian: 'Lithuanian',
norwegian: 'Norwegian',
persian: 'Persian',
portuguese: 'Portuguese',
romanian: 'Romanian',
russian: 'Russian',
sorani: 'Sorani (Kurdish)',
spanish: 'Spanish',
standard: 'Standard (generic)',
swedish: 'Swedish',
turkish: 'Turkish',
thai: 'Thai',
};
app.extensionData
.for('blomstra-search')
.for('sbp-search')
.registerSetting(function (this: any) {
// No index yet, or first build predates this tracking — stay silent.
const indexedAnalyzer = app.data.settings['blomstra-search.indexed-analyzer'];
if (!activeIndex || !indexedAnalyzer) return null;
const currentAnalyzer = this.setting('blomstra-search.analyzer-language')() || 'english';
const currentStemExclusion = this.setting('blomstra-search.stem-exclusion')() || '';
const indexedStemExclusion = app.data.settings['blomstra-search.indexed-stem-exclusion'] || '';
if (currentAnalyzer === indexedAnalyzer && currentStemExclusion === indexedStemExclusion) return null;
return m(
Alert,
{ type: 'warning', dismissible: false, icon: 'fas fa-exclamation-triangle' },
app.translator.trans('blomstra-search.admin.index-settings-changed')
);
})
.registerSetting({
setting: 'blomstra-search.elastic-endpoint',
label: app.translator.trans('blomstra-search.admin.elastic-endpoint'),
type: 'input'
type: 'input',
})
.registerSetting({
setting: 'blomstra-search.elastic-username',
label: app.translator.trans('blomstra-search.admin.elastic-username'),
type: 'input'
type: 'input',
})
.registerSetting({
setting: 'blomstra-search.elastic-password',
label: app.translator.trans('blomstra-search.admin.elastic-password'),
type: 'password'
type: 'password',
})
.registerSetting({
setting: 'blomstra-search.elastic-index',
label: app.translator.trans('blomstra-search.admin.elastic-index'),
default: 'flarum',
type: 'input'
type: 'input',
})
.registerSetting({
setting: 'blomstra-search.search-discussion-subjects',
label: app.translator.trans('blomstra-search.admin.search-discussion-subjects'),
type: 'switch',
})
.registerSetting({
setting: 'blomstra-search.search-post-bodies',
label: app.translator.trans('blomstra-search.admin.search-post-bodies'),
type: 'switch',
})
.registerSetting({
setting: 'blomstra-search.analyzer-language',
label: app.translator.trans('blomstra-search.admin.analyzer.label'),
help: app.translator.trans('blomstra-search.admin.analyzer.help'),
type: 'select',
options: Object.fromEntries(languages.entries()),
options: languages,
default: 'english',
})
.registerSetting({
setting: 'blomstra-search.elastic-index',
label: app.translator.trans('blomstra-search.admin.elastic-index'),
default: 'flarum',
type: 'input'
.registerSetting(function (this: any) {
const isCjk = (this.setting('blomstra-search.analyzer-language')() || 'english') === 'cjk';
return this.buildSettingComponent({
setting: 'blomstra-search.stem-exclusion',
type: 'textarea',
label: app.translator.trans('blomstra-search.admin.settings.stem-exclusion.label'),
help: app.translator.trans('blomstra-search.admin.settings.stem-exclusion.help'),
disabled: isCjk,
});
})
.registerSetting(function (this: any) {
const isCjk = (this.setting('blomstra-search.analyzer-language')() || 'english') === 'cjk';
return this.buildSettingComponent({
setting: 'blomstra-search.min-search-length',
label: app.translator.trans('blomstra-search.admin.min-search-length.label'),
help: app.translator.trans('blomstra-search.admin.min-search-length.help'),
type: 'select',
options: { '1': '1', '2': '2', '3': '3', '4': '4' },
default: app.data.settings['blomstra-search.min-search-length'],
disabled: isCjk,
});
});
});

View File

@ -1,15 +1,14 @@
import app from 'flarum/forum/app';
import {override} from 'flarum/common/extend';
import { override } from 'flarum/common/extend';
import DiscussionListState from 'flarum/forum/states/DiscussionListState';
export default function extendDiscussionState() {
override(DiscussionListState.prototype, 'loadPage', async function (this: DiscussionListState, original, page: number = 1) {
const preloaded = app.data.apiDocument || null;
// If existing payload is given or no search is made, fallback on native page.
if (preloaded || !this.requestParams()?.filter?.q) return original.call(this, page);
if (!this.requestParams()?.filter?.q) return original.call(this, page);
const params = this.requestParams();
params.page = {
@ -17,15 +16,16 @@ export default function extendDiscussionState() {
...params.page,
};
if (Array.isArray(params.include)) {
params.include = params.include.join(',');
}
// Construct API search URI
const url = `${app.forum.attribute('apiUrl')}/blomstra/search/${this.type}`;
const url = `${app.forum.attribute('apiUrl')}/sbp/search/${this.type}`;
// Make API GET request
const results = await app.request({params, url, method: 'GET'});
const results = await app.request({ params, url, method: 'GET' });
// Parse API response into models and push to store
return app.store.pushPayload(results);

View File

@ -21,18 +21,21 @@ export default class DiscussionsSearchSource implements SearchSource {
private type = 'discussions';
async search(query: string): Promise<void> {
// Suppress ES instant results when "search within comments" is ON;
// only usernames are shown in that mode.
query = query.toLowerCase();
this.results.set(query, []);
const params = {
filter: { q: query },
filter: { q: query, autocomplete: 1 },
page: { limit: 3 },
include: 'mostRelevantPost',
};
// Construct API search URI
const url = `${app.forum.attribute('apiUrl')}/blomstra/search/${this.type}`;
const url = `${app.forum.attribute('apiUrl')}/sbp/search/${this.type}`;
// Make API GET request
const results = await app.request({ params, url, method: 'GET' });
@ -44,7 +47,7 @@ export default class DiscussionsSearchSource implements SearchSource {
this.results.set(query, models);
}
view(query: string): Mithril.Children {
view(query: string): Array<Mithril.Vnode> {
query = query.toLowerCase();
// Get results from map
@ -54,7 +57,7 @@ export default class DiscussionsSearchSource implements SearchSource {
const mostRelevantPost = discussion.mostRelevantPost();
return (
<li className="DiscussionSearchResult" data-index={`${this.type}${discussion.id()}`} key={`${this.type}${discussion.id()}`}>
<li className="DiscussionSearchResult" data-index={`${this.type}${discussion.id()}`}>
<Link href={app.route.discussion(discussion, mostRelevantPost && mostRelevantPost.number())}>
<div className="DiscussionSearchResult-title">{highlight(discussion.title(), query)}</div>
{!!mostRelevantPost && <div className="DiscussionSearchResult-excerpt">{highlight(mostRelevantPost.contentPlain(), query, 100)}</div>}
@ -63,16 +66,14 @@ export default class DiscussionsSearchSource implements SearchSource {
);
});
return (
<>
<li className="Dropdown-header">{app.translator.trans('core.forum.search.discussions_heading')}</li>
return [
<li className="Dropdown-header">{app.translator.trans('core.forum.search.discussions_heading')}</li>,
<li>
<LinkButton icon="fas fa-search" href={app.route('index', { q: query })}>
{app.translator.trans('core.forum.search.all_discussions_button', { query })}
</LinkButton>
</li>
{results}
</>
);
</li>,
...results,
];
}
}

View File

@ -0,0 +1,26 @@
import { extend } from 'flarum/common/extend';
import DiscussionListItem from 'flarum/forum/components/DiscussionListItem';
import TerminalPost from 'flarum/forum/components/TerminalPost';
export default function extendDiscussionListItem() {
extend(DiscussionListItem.prototype, 'infoItems', function (items) {
const params = this.attrs.params;
if (!params.q) return;
const hasFieldSort = params.sort && params.sort !== 'relevance';
if (hasFieldSort) {
// Field sort active (latest, oldest, top, …): replace excerpt with TerminalPost.
// An excerpt is meaningless when results are ordered by date/count rather than relevance.
items.remove('excerpt');
if (!items.has('terminalPost')) {
items.add('terminalPost', <TerminalPost discussion={this.attrs.discussion} lastPost={!this.showFirstPost()} />);
}
} else if (!items.has('excerpt')) {
// Relevance mode but no excerpt (mostRelevantPost was null or non-comment type).
// Fall back to TerminalPost so the info section is never silently empty.
items.add('terminalPost', <TerminalPost discussion={this.attrs.discussion} lastPost={true} />);
}
});
}

View File

@ -1,20 +1,32 @@
import app from 'flarum/forum/app';
import Search from 'flarum/forum/components/Search';
import Search, { SearchAttrs, SearchSource } from 'flarum/forum/components/Search';
import { extend } from 'flarum/common/extend';
import ItemList from 'flarum/common/utils/ItemList';
import DiscussionsSearchSource from './SearchSources/DiscussionsSearchSource';
import extendDiscussionState from './PaginatedListStates/extendDiscussionState';
import extendDiscussionListItem from './extendDiscussionListItem';
app.initializers.add('blomstra-search', () => {
extend(Search.prototype, 'sourceItems', function (this: Search, items: ItemList) {
// items.remove('users');
items.remove('discussions');
extend(Search.prototype, 'sourceItems', function (this: Search<SearchAttrs>, items: ItemList<SearchSource>) {
// app.forum is not available during initializers (it is set after they run),
// so read the setting lazily here, at first render time.
const minLength = (app.forum.attribute('blomstraSearchMinLength') as number) || Search.MIN_SEARCH_LEN;
if (minLength !== Search.MIN_SEARCH_LEN) {
(Search as any).MIN_SEARCH_LEN = minLength;
}
items.add('discussions', new DiscussionsSearchSource());
items.replace('discussions', new DiscussionsSearchSource());
});
extendDiscussionState();
});
app.initializers.add(
'blomstra-search-early',
() => {
extendDiscussionState();
extendDiscussionListItem();
},
999999
);

View File

@ -0,0 +1,14 @@
diff --git a/src/Commands/BuildCommand.php b/src/Commands/BuildCommand.php
index 9024e1e..9024e1e 100644
--- a/src/Commands/BuildCommand.php
+++ b/src/Commands/BuildCommand.php
@@ -38,6 +38,6 @@ class BuildCommand extends Command
/** Matches Flarum's Search::MIN_SEARCH_LEN — the default minimum query length. */
public const DEFAULT_MIN_SEARCH_LENGTH = 3;
/** N-gram bounds for the title autocomplete subfield (not admin-configurable). */
- private const TITLE_MIN_GRAM = 2;
- private const TITLE_MAX_GRAM = 15;
+ private const TITLE_MIN_GRAM = 3;
+ private const TITLE_MAX_GRAM = 20;
protected $signature = 'blomstra:search:index

View File

@ -0,0 +1,3 @@
.ReindexWarningWidget {
padding: 0;
}

View File

@ -8,4 +8,21 @@ blomstra-search:
label: Analyzer language
help: |
The analyzer makes search understand stop words and undertakes language
specific improvements for indexing.
specific improvements for indexing. Changing this requires a full index rebuild to take effect.
index-settings-changed: "Search index settings have changed. Run: php flarum blomstra:search:index build"
reindex-required:
title: Search index rebuild required
detail: "The search index is not compatible with this version of the extension. Run: php flarum blomstra:search:index build"
search-discussion-subjects: Search inside discussion titles
search-post-bodies: Search inside comments
match-sentences: Match search term against full sentence
match-words: Match search term against full words
settings:
stem-exclusion:
label: "Protected words (stem exclusion)"
help: "One word per line. These words will not be stemmed during search (e.g. brand names, proper nouns). Not applicable for CJK or Thai languages. Requires a full re-index to take effect."
min-search-length:
label: Minimum search query length
help: |
Minimum number of characters required before a search is triggered. Lower this to 1 or 2
for CJK (Chinese, Japanese, Korean) communities where single characters carry full meaning.

View File

@ -1,6 +1,16 @@
<?php
namespace Blomstra\Search\Api;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Api;
use Illuminate\Support\Arr;
use Psr\Http\Message\ResponseInterface;
@ -9,7 +19,9 @@ class Client extends \Flarum\Api\Client
{
public function get(string $path): ResponseInterface
{
if ($path === '/discussions' && Arr::has($this->queryParams, 'filter.q')) return parent::get("/blomstra/search/discussions");
if ($path === '/discussions' && Arr::has($this->queryParams, 'filter.q')) {
return parent::get('/blomstra/search/discussions');
}
return parent::get($path);
}

View File

@ -1,8 +1,18 @@
<?php
namespace Blomstra\Search\Api\Controllers;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
use Blomstra\Search\Commands\BuildCommand;
namespace Sbp\Search\Api\Controllers;
use Sbp\Search\Commands\BuildCommand;
use Laminas\Diactoros\Response\EmptyResponse;
use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\ServerRequestInterface;
@ -23,7 +33,8 @@ class IndexController implements RequestHandlerInterface
$command->run(
new ArrayInput([]),
new ConsoleOutput);
new ConsoleOutput()
);
return new EmptyResponse();
}

View File

@ -1,11 +1,24 @@
<?php
namespace Blomstra\Search\Api\Controllers;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
use Blomstra\Search\Elasticsearch\MatchPhraseQuery;
use Blomstra\Search\Elasticsearch\MatchQuery;
use Blomstra\Search\Save\Document as ElasticDocument;
use Blomstra\Search\Elasticsearch\TermsQuery;
namespace Sbp\Search\Api\Controllers;
use Sbp\Search\Elasticsearch\HasChildQuery;
use Sbp\Search\Elasticsearch\MatchPhraseQuery;
use Sbp\Search\Elasticsearch\MatchQuery;
use Sbp\Search\Elasticsearch\TermsQuery;
use Sbp\Search\Searchers\CommentPostSearcher;
use Sbp\Search\Searchers\DiscussionSearcher;
use Sbp\Search\Searchers\Searcher;
use Elasticsearch\Client;
use Flarum\Api\Controller\ListDiscussionsController;
use Flarum\Api\Serializer\DiscussionSerializer;
@ -14,15 +27,17 @@ use Flarum\Extension\ExtensionManager;
use Flarum\Group\Group;
use Flarum\Http\RequestUtil;
use Flarum\Http\UrlGenerator;
use Flarum\Settings\SettingsRepositoryInterface;
use Flarum\Tags\Tag;
use Flarum\User\User;
use Illuminate\Contracts\Container\Container;
use Illuminate\Support\Arr;
use Illuminate\Support\Collection;
use Illuminate\Support\Str;
use Psr\Http\Message\ServerRequestInterface;
use Psr\Log\LoggerInterface;
use Spatie\ElasticsearchQueryBuilder\Builder;
use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
use Sbp\Search\Elasticsearch\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery;
use Spatie\ElasticsearchQueryBuilder\Sorts\Sort;
use Tobscure\JsonApi\Document;
@ -34,92 +49,134 @@ class SearchController extends ListDiscussionsController
protected array $translateSort = [
'lastPostedAt' => 'updated_at',
'createdAt' => 'created_at',
'commentCount' => 'comment_count'
'commentCount' => 'comment_count',
'view_count' => 'view_count',
];
public function __construct(protected Client $elastic, protected UrlGenerator $uri)
{}
protected bool $matchSentences;
protected bool $matchWords;
protected ?Searcher $discussionSearcher;
protected ?Searcher $postSearcher;
public function __construct(protected Client $elastic, protected UrlGenerator $uri, Container $container, SettingsRepositoryInterface $settings)
{
$this->matchSentences = true;
$this->matchWords = true;
$searchers = collect($container->tagged('blomstra.search.searchers'));
$this->discussionSearcher = $searchers->first(fn ($s) => $s instanceof DiscussionSearcher);
$this->postSearcher = $searchers->first(fn ($s) => $s instanceof CommentPostSearcher);
}
protected function data(ServerRequestInterface $request, Document $document)
{
// Not used for now.
$type = Arr::get($request->getQueryParams(), 'type');
$actor = RequestUtil::getActor($request);
$filters = $this->extractFilter($request);
$search = $this->getSearch($filters);
$limit = $this->extractLimit($request);
$offset = $this->extractOffset($request);
$include = array_merge($this->extractInclude($request), ['state']);
$filterQuery = BoolQuery::create();
if (! empty($search)) {
$filterQuery
// @todo commented out to use only partial matching for now
->add($this->sentenceMatch($search))
->add($this->wordMatch($search, 'and'))
->add($this->wordMatch($search, 'or'))
// ->add($this->partialMatch($search))
;
// Autocomplete path: fast prefix match on title.autocomplete only.
if (!empty($filters['autocomplete']) && !empty($search)) {
return $this->handleAutocomplete($search, $actor, $limit, $offset, $include, $document, $request);
}
$knownSortFields = array_merge(array_values($this->translateSort), ['rawId']);
$logger = resolve(LoggerInterface::class);
$phpSortField = null;
$phpSortDir = 'desc';
$needsScoring = true;
$sorts = [];
foreach ($this->extractSort($request) as $field => $direction) {
$translated = $this->translateSort[$field] ?? $field;
if (!in_array($translated, $knownSortFields)) {
$logger->warning("blomstra/search: unknown sort field \"{$field}\", ignoring.");
continue;
}
$sorts[] = new Sort($translated, $direction);
$needsScoring = false;
if ($phpSortField === null && $translated !== 'rawId') {
$phpSortField = $translated;
$phpSortDir = $direction;
}
}
// Default to latest when no explicit sort is requested. This lets has_child use
// score_mode:none, which skips child scoring entirely and lets ES short-circuit
// early on large corpora. Relevance is still available via sort=relevant if added.
if (empty($sorts)) {
$needsScoring = false;
$phpSortField = 'updated_at';
$phpSortDir = 'desc';
$sorts[] = new Sort('updated_at', 'desc');
}
$query = BoolQuery::create()
// Always restrict to discussion documents; posts are only searched via has_child.
->add(TermQuery::create('join_field', 'discussion'), 'filter');
if (!empty($search)) {
$query->add($this->buildTextQuery($search, $actor, $needsScoring));
}
$this->addFilters($query, $actor, $filters);
$builder = (new Builder($this->elastic))
->index(resolve('blomstra.search.elastic_index'))
->size($limit + 1)
->from($this->extractOffset($request))
->addQuery(
$this->addFilters($filterQuery, $actor, $filters)
);
->addQuery($query);
foreach ($this->extractSort($request) as $field => $direction) {
$field = $this->translateSort[$field] ?? $field;
$builder->addSort(new Sort($field, $direction));
foreach ($sorts as $sort) {
$builder->addSort($sort);
}
$response = $builder->search();
// track_total_hits: false lets ES stop counting once it has collected
// enough results in sort order, avoiding a full-index count on every query.
$payload = $builder->getPayload();
$payload['track_total_hits'] = false;
$response = $this->elastic->search([
'index' => resolve('blomstra.search.elastic_index'),
'size' => $limit + 1,
'from' => $offset,
'body' => $payload,
]);
Discussion::setStateUser($actor);
// Eager load groups for use in the policies (isAdmin check)
if (in_array('mostRelevantPost.user', $include)) {
$include[] = 'mostRelevantPost.user.groups';
// If the first level of the relationship wasn't explicitly included,
// add it so the code below can look for it
if (! in_array('mostRelevantPost', $include)) {
if (!in_array('mostRelevantPost', $include)) {
$include[] = 'mostRelevantPost';
}
}
// we need to retrieve all discussion ids and when the results are posts,
// their ids as most relevant post id
// All hits are discussion documents. Extract the best-matching post ID from
// inner_hits when a has_child clause matched (i.e. the match came from a post).
$results = Collection::make(Arr::get($response, 'hits.hits'))
->map(function ($hit) {
$type = $hit['_source']['type'];
$id = Str::after($hit['_source']['id'], "$type:");
// _id is "discussions:123" — parse the numeric part directly.
$discussionId = Str::after($hit['_id'], 'discussions:');
// rawId on the inner hit gives us the integer post ID.
$bestPostId = Arr::get($hit, 'inner_hits.best_post.hits.hits.0._source.rawId');
if ($type === 'posts') {
return [
'most_relevant_post_id' => $id,
'weight' => Arr::get($hit, 'sort.0')
'discussion_id' => $discussionId,
'most_relevant_post_id' => $bestPostId,
'weight' => Arr::get($hit, 'sort.0', Arr::get($hit, '_score', 0)),
];
} else {
return [
'discussion_id' => $id,
'weight' => Arr::get($hit, 'sort.0')
];
}
});
$document->addPaginationLinks(
$this->uri->to('api')->route('blomstra.search', [
'type' => 'discussions'
]),
$this->uri->to('api')->route('sbp.search', ['type' => 'discussions']),
$request->getQueryParams(),
$offset,
$limit,
@ -129,23 +186,25 @@ class SearchController extends ListDiscussionsController
$results = $results->take($limit);
$discussions = Discussion::query()
->select('discussions.*')
->join('posts', 'posts.discussion_id', 'discussions.id')
->whereIn('discussions.id', $results->pluck('discussion_id')->filter())
->orWhereIn('posts.id', $results->pluck('most_relevant_post_id')->filter())
->when(
$actor->isGuest() || !$actor->hasPermission('discussion.hide'),
fn ($q) => $q->whereNull('hidden_at')
)
->whereIn('id', $results->pluck('discussion_id')->filter())
->get()
->each(function (Discussion $discussion) use ($results) {
if (in_array($discussion->id, $results->pluck('discussion_id')->toArray())) {
$discussion->most_relevant_post_id = $discussion->first_post_id;
$discussion->weight = $results->firstWhere('discussion_id', $discussion->id)['weight'] ?? 0;
} else {
$post = $discussion->posts()->whereIn('id', $results->pluck('most_relevant_post_id'))->first();
$discussion->most_relevant_post_id = $post?->id ?? $discussion->first_post_id;
$discussion->weight = $results->firstWhere('most_relevant_post_id', $post?->id)['weight'] ?? 0;
}
$result = $results->firstWhere('discussion_id', $discussion->id);
$discussion->most_relevant_post_id = $result['most_relevant_post_id']
?? $discussion->first_post_id;
$discussion->weight = $result['weight'] ?? 0;
})
->keyBy('id')
->sortByDesc('weight')
->when(
$phpSortField,
fn ($c) => $phpSortDir === 'desc' ? $c->sortByDesc($phpSortField) : $c->sortBy($phpSortField),
fn ($c) => $c->sortByDesc('weight')
)
->unique();
$this->loadRelations($discussions, $include);
@ -163,33 +222,208 @@ class SearchController extends ListDiscussionsController
return $discussions;
}
protected function getDocument(string $type): ?ElasticDocument
{
$documents = resolve(Container::class)->tagged('blomstra.search.documents');
/**
* Fast autocomplete path: match prefix n-grams on title.autocomplete only.
* No has_child, no phrase scoring a single MatchQuery suffices.
*/
protected function handleAutocomplete(
string $search,
User $actor,
int $limit,
int $offset,
array $include,
Document $document,
ServerRequestInterface $request
): mixed {
$query = BoolQuery::create()
->add(TermQuery::create('join_field', 'discussion'), 'filter');
return collect($documents)->first(function (ElasticDocument $document) use ($type) {
return $document->type() === $type;
});
$query->add($this->buildAutocompleteQuery($search), 'must');
$this->addFilters($query, $actor, []);
$payload = (new Builder($this->elastic))
->index(resolve('blomstra.search.elastic_index'))
->addQuery($query)
->addSort(new Sort('updated_at', 'desc'))
->getPayload();
$response = $this->elastic->search([
'index' => resolve('blomstra.search.elastic_index'),
'size' => $limit + 1,
'from' => $offset,
'body' => $payload,
]);
Discussion::setStateUser($actor);
$results = Collection::make(Arr::get($response, 'hits.hits'))
->map(fn ($hit) => [
'discussion_id' => Str::after($hit['_id'], 'discussions:'),
'most_relevant_post_id' => null,
'weight' => Arr::get($hit, 'sort.0', 0),
]);
$document->addPaginationLinks(
$this->uri->to('api')->route('sbp.search', ['type' => 'discussions']),
$request->getQueryParams(),
$offset,
$limit,
$results->count() > $limit ? null : 0
);
$results = $results->take($limit);
$discussions = Discussion::query()
->when(
$actor->isGuest() || !$actor->hasPermission('discussion.hide'),
fn ($q) => $q->whereNull('hidden_at')
)
->whereIn('id', $results->pluck('discussion_id')->filter())
->get()
->each(function (Discussion $discussion) use ($results) {
$result = $results->firstWhere('discussion_id', $discussion->id);
$discussion->most_relevant_post_id = $result['most_relevant_post_id'] ?? $discussion->first_post_id;
$discussion->weight = $result['weight'] ?? 0;
})
->sortByDesc('updated_at')
->unique();
$this->loadRelations($discussions, $include);
return $discussions;
}
protected function buildAutocompleteQuery(string $search): BoolQuery
{
$query = BoolQuery::create();
// Title prefix match (stronger)
$query->add(
(new MatchQuery('title.autocomplete', $search))
->operator('and')
->boost(1.0),
'should'
);
// Content prefix match (weaker)
$query->add(
(new MatchQuery('content.autocomplete', $search))
->operator('and')
->boost(0.4),
'should'
);
$query->minimumShouldMatch(1);
return $query;
}
/**
* Build the text-matching portion of the query.
*
* Discussion titles are matched directly (filtered to join_field=discussion).
* Post bodies are matched via has_child. When $needsScoring is true (no explicit
* sort, i.e. relevance ordering), score_mode=sum accumulates child scores onto the
* parent so that discussions with many strongly-matching posts rank higher. When
* false (any explicit field sort), score_mode=none skips scoring entirely ES only
* checks whether a matching child exists, which is significantly cheaper on large
* corpora.
* inner_hits returns the best-matching post for use as mostRelevantPost.
* Hidden posts are only included in matching for users with post.hide permission.
*/
protected function buildTextQuery(string $search, User $actor, bool $needsScoring = false): BoolQuery
{
$textQuery = BoolQuery::create();
// Strong direct title match for the full search page.
$textQuery->add(
(new MatchQuery('title', $search))
->operator('and')
->boost(4.0),
'should'
);
// Low-weight prefix match on title.autocomplete for partial title hits.
$textQuery->add(
(new MatchQuery('title.autocomplete', $search))
->operator('and')
->boost(0.35),
'should'
);
// Low-weight prefix match on title.autocomplete for the full search page.
// Exact / normal title matches should still rank higher than prefix-only hits.
$textQuery->add(
(new MatchQuery('title.autocomplete', $search))
->operator('and')
->boost(0.35),
'should'
);
if ($this->discussionSearcher?->enabled()) {
$textQuery->add($this->buildShouldClauses($search, $this->discussionSearcher->boost()), 'should');
}
if ($this->postSearcher?->enabled()) {
$postQuery = $this->buildShouldClauses($search, $this->postSearcher->boost());
// Guests and non-moderators may not see hidden posts; exclude them from child matching.
if ($actor->isGuest() || !$actor->hasPermission('post.hide')) {
$postQuery->add(TermQuery::create('is_hidden', 'false'), 'filter');
}
// Without minimum_should_match, ES default MSM is 0 when a filter clause is present,
// causing has_child to score every non-hidden post instead of only matching ones.
$postQuery->minimumShouldMatch(1);
$textQuery->add(
HasChildQuery::create('post', $postQuery, $needsScoring ? 'sum' : 'none')->withInnerHits(),
'should'
);
}
return $textQuery;
}
protected function buildShouldClauses(string $search, float $boost): BoolQuery
{
$should = BoolQuery::create();
if ($this->matchSentences) {
$should->add((new MatchPhraseQuery('content', $search))->boost(2 * $boost), 'should');
}
if ($this->matchWords) {
$should->add((new MatchQuery('content', $search))->operator('and')->boost(1.8 * $boost), 'should');
$should->add((new MatchQuery('content.autocomplete', $search))->operator('and')->boost(0.30 * $boost), 'should');
}
return $should;
}
protected function extensionEnabled(string $extension): bool
{
/** @var ExtensionManager $manager */
$manager = resolve(ExtensionManager::class);
return $manager->isEnabled($extension);
return resolve(ExtensionManager::class)->isEnabled($extension);
}
protected function addFilters(BoolQuery $query, User $actor, array $filters = []): BoolQuery
protected function addFilters(BoolQuery $query, User $actor, array $filters = []): void
{
$groups = $this->getGroups($actor);
$onlyPrivate = Str::contains($filters['q'] ?? '', 'is:private');
$subQuery = BoolQuery::create()
->add(TermQuery::create('is_private', 'false'))
->add(TermsQuery::create('groups', $groups->toArray()));
if ($this->extensionEnabled('flarum-tags') && !empty($filters['tag'])) {
$slugs = is_array($filters['tag']) ? $filters['tag'] : explode(',', $filters['tag']);
$tagIds = Tag::query()->whereIn('slug', $slugs)->pluck('id')->toArray();
if (!empty($tagIds)) {
$query->add(TermsQuery::create('tags', $tagIds), 'filter');
}
}
if ($this->extensionEnabled('fof-byobu') && $actor->exists) {
$byobuQuery = BoolQuery::create()
->add(TermQuery::create('is_private', 'true'))
@ -208,86 +442,12 @@ class SearchController extends ListDiscussionsController
}
}
$query->add(
$subQuery,
'filter'
);
return $query;
$query->add($subQuery, 'filter');
}
protected function sentenceMatch(string $q): Query
{
$query = (new MatchPhraseQuery('content', $q));
return BoolQuery::create()
// Discussion titles
->add(
BoolQuery::create()
->add(TermQuery::create('type', 'discussions'), 'filter')
->add($query->boost(2)),
'should'
)
// Post bodies
->add(
BoolQuery::create()
->add(TermQuery::create('type', 'posts'), 'filter')
->add($query->boost(1.9)),
'should'
);
}
protected function wordMatch(string $q, string $operator = 'or')
{
$query = (new MatchQuery('content', $q))
->operator($operator);
$boost = $operator === 'and' ? 1 : .8;
return BoolQuery::create()
// Discussion titles
->add(
BoolQuery::create()
->add(TermQuery::create('type', 'discussions'), 'filter')
->add($query->boost($boost * 1.8)),
'should'
)
// Post bodies
->add(
BoolQuery::create()
->add(TermQuery::create('type', 'posts'), 'filter')
->add($query->boost($boost * 1.8)),
'should'
);
}
protected function partialMatch(string $q)
{
$query = (new MatchQuery('content', $q));
return BoolQuery::create()
// Discussion titles
->add(
BoolQuery::create()
->add(TermQuery::create('type', 'discussions'), 'filter')
->add(clone $query->boost(1.6)),
'should'
)
// Post bodies
->add(
BoolQuery::create()
->add(TermQuery::create('type', 'posts'), 'filter')
->add(clone $query->boost(1.3)),
'should'
);
}
protected function getGroups(User $actor): Collection
{
/** @var Collection $groups */
$groups = $actor->groups->pluck('id');
$groups->add(Group::GUEST_ID);
if ($actor->is_email_confirmed) {
@ -302,10 +462,12 @@ class SearchController extends ListDiscussionsController
$search = Arr::get($filters, 'q');
if ($search) {
// Strip Flarum gambit operators (tag:foo, author:bar, is:private, etc.)
// before passing to ES. These are structural filters handled separately;
// leaving them in causes operator:and to require the gambit tokens to
// appear literally in post content, producing zero results.
$q = collect(explode(' ', $search))
->filter(function (string $part) {
return $part !== 'is:private';
})
->filter(fn (string $part) => !preg_match('/^\w+:/', $part))
->filter()
->join(' ');

View File

@ -1,143 +1,398 @@
<?php
namespace Blomstra\Search\Commands;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
use Blomstra\Search\Jobs\Job;
use Blomstra\Search\Jobs\SavingJob;
use Blomstra\Search\Seeders\Seeder;
use Carbon\Carbon;
namespace Sbp\Search\Commands;
use Sbp\Search\Jobs\Job;
use Sbp\Search\Jobs\UpdateSearchJob;
use Sbp\Search\Seeders\Seeder;
use Elasticsearch\Client;
use Elasticsearch\Common\Exceptions\ElasticsearchException;
use Flarum\Settings\SettingsRepositoryInterface;
use Illuminate\Console\Command;
use Illuminate\Contracts\Container\Container;
use Illuminate\Contracts\Queue\Queue;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Support\Arr;
use Spatie\ElasticsearchQueryBuilder\Builder;
use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\RangeQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery;
class BuildCommand extends Command
{
/**
* Bump this when a mapping change requires a full reindex.
* Written into the mapping's _meta.index_compat_version on every build and
* read back into blomstra-search.index-compatible by saveIndexedConfig so the
* value always reflects the live index including after a promote/rollback.
*/
public const INDEX_COMPAT_VERSION = 'v3';
/** Matches Flarum's Search::MIN_SEARCH_LEN — the default minimum query length. */
public const DEFAULT_MIN_SEARCH_LENGTH = 3;
/** N-gram bounds for the title autocomplete subfield (not admin-configurable). */
private const TITLE_MIN_GRAM = 3;
private const TITLE_MAX_GRAM = 20;
protected $signature = 'blomstra:search:index
{--max-id= : Limits for each object the number of items to seed}
{--chunk-size= : Size of the chunks to dispatch into jobs}
{--throttle= : Number of seconds to wait between pushing to the queue}
{--only= : type to run seeder for, eg discussions or posts}
{--recreate : create or recreate the index}
{--mapping : recreate the mapping}
{--continue : continue each object type where you left off}';
protected $description = 'Rebuilds the complete search server with its documents.';
{action? : build | promote | rollback | discard | mapping | fill}
{--fresh}
{--resume}
{--staging}
{--keep-backup}
{--pending}
{--backup}
{--only=}
{--max-id=}
{--throttle=}
{--i-am-sure}';
public function handle(Container $container)
protected $description = 'Build and manage the Elasticsearch search index.';
protected $help = <<<'HELP'
<comment>Actions and their options:</comment>
<info>build</info> Queue documents and promote automatically.
<comment>--resume</comment> Resume an interrupted build from where it left off
<comment>--fresh</comment> Drop the staging index and start completely fresh
<comment>--staging</comment> Keep in staging requires explicit <info>promote</info> (blue-green)
<comment>--keep-backup</comment> Retain the replaced index for rollback
<info>promote</info> Swap alias to staging index (blue-green workflow).
<comment>--keep-backup</comment> Retain the replaced index for rollback
<comment>--i-am-sure</comment> Skip the confirmation prompt
<info>rollback</info> Restore the backup index to live.
<info>discard</info> Drop an index without promoting.
<comment>--pending</comment> Drop the staging index (cancel a build)
<comment>--backup</comment> Drop the backup index (cleanup after --keep-backup)
<info>mapping</info> Push updated mapping to the live index only.
<info>fill</info> Seed only documents missing from the live index.
<comment>Shared seeding options (build / fill):</comment>
<comment>--only=TYPE</comment> Seed only this type: <info>discussions</info> or <info>posts</info>
<comment>--throttle=N</comment> Seconds to wait between batches
<comment>--max-id=N</comment> Limit seeding to IDs up to this value
HELP;
public function handle(Container $container): void
{
$index = $container->make('blomstra.search.elastic_index');
/** @var array $seeders */
$seeders = $container->tagged('blomstra.search.seeders');
/** @var Queue $queue */
$queue = $container->make(Queue::class);
/** @var Client $client */
$client = $container->make(Client::class);
/** @var SettingsRepositoryInterface $settings */
$settings = $container->make(SettingsRepositoryInterface::class);
$properties = [
'properties' => [
'content' => ['type' => 'text', 'analyzer' => 'flarum_analyzer_partial', 'search_analyzer' => 'flarum_analyzer'],
'created_at' => ['type' => 'date'],
'updated_at' => ['type' => 'date'],
'is_private' => ['type' => 'boolean'],
'is_sticky' => ['type' => 'boolean'],
'groups' => ['type' => 'integer'],
'recipient_groups' => ['type' => 'integer'],
'recipient_users' => ['type' => 'integer'],
'comment_count' => ['type' => 'integer'],
]
];
/** @var string $alias */
$alias = $container->make('blomstra.search.elastic_index');
if ($this->option('recreate')) {
// Flush the index.
$client->indices()->delete([
'index' => $index,
'ignore_unavailable' => true
]);
// Create a new index.
$client->indices()->create([
'index' => $index,
'body' => [
'settings' => [
'index.max_ngram_diff' => 10,
'analysis' => [
'analyzer' => [
'flarum_analyzer' => [
'type' => $settings->get('blomstra-search.analyzer-language') ?: 'english'
],
'flarum_analyzer_partial' => [
'type' => 'custom',
'tokenizer' => 'standard',
'filter' => [
'lowercase',
'partial_search_filter'
]
]
],
'filter' => [
'partial_search_filter' => [
'type' => 'ngram',
'min_gram' => 1,
'max_gram' => 10,
'token_chars' => ['letter', 'digit', 'symbol']
]
]
]
]
]
]);
if (!$this->argument('action')) {
$this->call('help', ['command_name' => $this->getName()]);
return;
}
if ($this->option('recreate') || $this->option('mapping')) {
switch ($this->argument('action')) {
case 'build':
$this->runBuild($client, $alias, $settings, $container);
break;
case 'promote':
$this->runPromote($client, $alias, $settings);
break;
case 'rollback':
$this->runRollback($client, $alias, $settings);
break;
case 'discard':
$this->runDiscard($client, $settings);
break;
case 'mapping':
$client->indices()->putMapping([
'index' => $index,
'body' => $properties
'index' => $alias,
'body' => $this->mappingProperties(),
]);
$this->info('Mapping updated on live index.');
break;
case 'fill':
$this->runSeeders(
collect($container->tagged('blomstra.search.seeders')),
$container->make(Queue::class),
$client,
$settings,
$alias,
seedMissing: true
);
break;
default:
$this->error("Unknown action '{$this->argument('action')}'. Valid actions: build, promote, rollback, discard, mapping, fill.");
}
}
protected function runBuild(Client $client, string $alias, SettingsRepositoryInterface $settings, Container $container): void
{
/** @var Seeder[] $seeders */
$seeders = collect($container->tagged('blomstra.search.seeders'));
// Guard: staging build exists without explicit intent given.
$staging = $settings->get('blomstra-search.staging-index');
if ($staging && $client->indices()->exists(['index' => $staging])
&& !$this->option('resume') && !$this->option('fresh')
) {
$this->warn("An in-progress build already exists: $staging");
$this->line('');
$this->line('Choose one of:');
$this->line(' blomstra:search:index build --resume Continue from where it left off');
$this->line(' blomstra:search:index build --resume --staging Continue and keep in staging when done');
$this->line(' blomstra:search:index build --fresh Drop this build and start completely fresh');
$this->line(' blomstra:search:index build --fresh --staging Start fresh and keep in staging when done');
return;
}
$aliasExists = (bool) $client->indices()->existsAlias(['name' => $alias]);
$indexExists = !$aliasExists && (bool) $client->indices()->exists(['index' => $alias]);
if (!$aliasExists && !$indexExists) {
// First install: create a timestamped index, alias it immediately, seed live.
$targetIndex = $this->prepareFirstInstall($client, $alias, $settings, $seeders);
$stagingBuild = false;
} else {
// Live system: build into a staging index without touching the live alias.
$targetIndex = $this->prepareStagingIndex($client, $alias, $settings, $seeders);
$stagingBuild = true;
}
$client->indices()->putMapping([
'index' => $targetIndex,
'body' => $this->mappingProperties(),
]);
$this->runSeeders($seeders, $container->make(Queue::class), $client, $settings, $targetIndex);
if ($stagingBuild) {
if ($this->option('staging')) {
$staging = $settings->get('blomstra-search.staging-index');
$this->info("Staging build complete. Drain the queue, then promote '$staging' to live:");
$this->line(' php flarum queue:work --stop-when-empty');
$this->line(' php flarum blomstra:search:index promote');
} else {
$this->info('Jobs queued. Promoting now (search improves as the queue drains)...');
$this->performPromote($client, $alias, $settings);
}
}
}
protected function runPromote(Client $client, string $alias, SettingsRepositoryInterface $settings): void
{
if (!$this->option('i-am-sure')) {
$this->warn('promote switches the live index. Ensure all queued jobs have finished first:');
$this->line(' php flarum queue:work --stop-when-empty');
$this->line('');
if (!$this->confirm('Queue drained? Proceed?')) {
return;
}
}
$this->performPromote($client, $alias, $settings);
}
protected function performPromote(Client $client, string $alias, SettingsRepositoryInterface $settings): void
{
$staging = $settings->get('blomstra-search.staging-index');
if (!$staging || !$client->indices()->exists(['index' => $staging])) {
$this->error('No staging index ready to promote. Run: blomstra:search:index build');
return;
}
$aliasExists = (bool) $client->indices()->existsAlias(['name' => $alias]);
$indexExists = !$aliasExists && (bool) $client->indices()->exists(['index' => $alias]);
if ($aliasExists) {
$result = $client->indices()->getAlias(['name' => $alias]);
$activeIndex = array_key_first($result);
$client->indices()->updateAliases([
'body' => ['actions' => [
['remove' => ['index' => $activeIndex, 'alias' => $alias]],
['add' => ['index' => $staging, 'alias' => $alias]],
]],
]);
$this->info("'$alias' now points to '$staging'. Promote complete.");
if ($this->option('keep-backup')) {
$settings->set('blomstra-search.backup-index', $activeIndex);
$this->info("Kept '$activeIndex' as backup — run rollback to revert, discard --backup to drop it.");
} else {
$client->indices()->delete(['index' => $activeIndex, 'ignore_unavailable' => true]);
$this->info("Deleted old index: $activeIndex");
}
} else {
// One-time migration: concrete index → alias (legacy installs only).
if ($indexExists) {
$client->indices()->delete(['index' => $alias]);
$this->info("Deleted legacy concrete index: $alias");
}
$client->indices()->putAlias(['index' => $staging, 'name' => $alias]);
$this->info("'$alias' now points to '$staging'. Promote complete.");
}
$settings->set('blomstra-search.active-index', $staging);
$settings->set('blomstra-search.staging-index', null);
$this->saveIndexedConfig($client, $settings, $staging);
}
protected function runRollback(Client $client, string $alias, SettingsRepositoryInterface $settings): void
{
$backup = $settings->get('blomstra-search.backup-index');
if (!$backup || !$client->indices()->exists(['index' => $backup])) {
$this->error('No backup index to roll back to. Was promote run with --keep-backup?');
return;
}
$result = $client->indices()->getAlias(['name' => $alias]);
$activeIndex = array_key_first($result);
$client->indices()->updateAliases([
'body' => ['actions' => [
['remove' => ['index' => $activeIndex, 'alias' => $alias]],
['add' => ['index' => $backup, 'alias' => $alias]],
]],
]);
$client->indices()->delete(['index' => $activeIndex, 'ignore_unavailable' => true]);
$this->info("Rolled back: '$alias' now points to '$backup'. Deleted '$activeIndex'.");
$settings->set('blomstra-search.active-index', $backup);
$settings->set('blomstra-search.backup-index', null);
$this->saveIndexedConfig($client, $settings, $backup);
}
protected function runDiscard(Client $client, SettingsRepositoryInterface $settings): void
{
$pending = $this->option('pending');
$backup = $this->option('backup');
if (!$pending && !$backup) {
$this->error('Specify what to discard: --pending (staging build) or --backup (rollback index).');
return;
}
if ($pending) {
$staging = $settings->get('blomstra-search.staging-index');
if (!$staging) {
$this->warn('No staging index to discard.');
} else {
if ($client->indices()->exists(['index' => $staging])) {
$client->indices()->delete(['index' => $staging]);
}
$settings->set('blomstra-search.staging-index', null);
$this->info("Discarded staging index: $staging");
}
}
if ($backup) {
$backupIndex = $settings->get('blomstra-search.backup-index');
if (!$backupIndex) {
$this->warn('No backup index to discard.');
} else {
if ($client->indices()->exists(['index' => $backupIndex])) {
$client->indices()->delete(['index' => $backupIndex]);
}
$settings->set('blomstra-search.backup-index', null);
$this->info("Discarded backup index: $backupIndex");
}
}
}
protected function runSeeders(
iterable $seeders,
Queue $queue,
Client $client,
SettingsRepositoryInterface $settings,
string $targetIndex,
bool $seedMissing = false
): void {
$only = $this->option('only');
/** @var Seeder $seeder */
foreach ($seeders as $seeder) {
if ($only && $seeder->type() !== $only) continue;
if ($only && $seeder->type() !== $only) {
continue;
}
$total = 0;
$continueAt = $this->option('continue')
? ($this->continueAt($seeder->type()) ?? $seeder->query()->max('id'))
: $seeder->query()->max('id');
if ($this->option('resume')) {
$saved = $this->getContinueAt($settings, $seeder->type());
if ($saved === 0) {
$this->info("Seeder '{$seeder->type()}' already completed in a previous run, skipping.");
continue;
}
$continueAt = $saved ?? $seeder->query()->max('id');
} else {
$continueAt = $seeder->query()->max('id');
}
$seeded = null;
while ($continueAt !== null) {
$rangeFrom = max(1, $continueAt - 2500);
$rangeTo = $continueAt;
if ($seedMissing) {
$seeded = $this->queryIndexedIds($client, $targetIndex, $seeder->joinRelation(), $rangeFrom, $rangeTo);
}
while($continueAt !== null) {
/** @var Collection $collection */
$collection = $seeder->query()
->latest('id')
->whereBetween('id', [$continueAt - 1000, $continueAt])
->when($this->option('max-id'), function ($query, $id) {
$query->where('id', '<=', $id);
})
->whereBetween('id', [$rangeFrom, $rangeTo])
->when($this->option('max-id'), fn ($q, $id) => $q->where('id', '<=', $id))
->when($seeded, fn ($q, $seeded) => $q->whereNotIn('id', $seeded))
->get();
$min = $collection->min('id');
if ($seedMissing && $collection->isEmpty()) {
$continueAt = $rangeFrom > 2 ? $rangeFrom - 1 : null;
} else {
$continueAt = $min && $min > 2 ? $min - 1 : null;
}
$queue->pushOn(Job::$onQueue, new SavingJob($collection, $seeder));
if ($collection->isNotEmpty()) {
$queue->pushOn(Job::$onQueue, new UpdateSearchJob($collection, $seeder, $targetIndex));
}
$this->info("Pushed into the index, type: {$seeder->type()}, amount: {$collection->count()}.");
$this->info("IDs {$rangeFrom}{$rangeTo} | type: {$seeder->type()} | queued: {$collection->count()}.");
$total += $collection->count();
$this->continueAt(
$seeder->type(),
$continueAt
);
$this->setContinueAt($settings, $seeder->type(), $continueAt);
if ($throttle = $this->option('throttle')) {
$this->info("Throttling for $throttle seconds");
@ -145,21 +400,274 @@ class BuildCommand extends Command
}
}
$this->info("Pushed a total of $total into the index.");
$this->setContinueAt($settings, $seeder->type(), 0);
$this->info("Queued a total of $total {$seeder->type()} for indexing.");
}
}
protected function continueAt(string $type, int $at = null)
/**
* First install: create a timestamped concrete index, alias the configured name to it
* immediately, and return the alias so seeding goes through it and is live from the start.
*/
protected function prepareFirstInstall(
Client $client,
string $alias,
SettingsRepositoryInterface $settings,
iterable $seeders
): string {
$concrete = $alias . '_' . date('YmdHis');
$client->indices()->create([
'index' => $concrete,
'body' => ['settings' => $this->buildIndexSettings($settings)],
]);
$client->indices()->putAlias(['index' => $concrete, 'name' => $alias]);
$settings->set('blomstra-search.active-index', $concrete);
$this->saveIndexedConfig($client, $settings, $concrete);
foreach ($seeders as $seeder) {
$this->setContinueAt($settings, $seeder->type(), null);
}
$this->info("Created '$concrete', aliased '$alias' → '$concrete'.");
$this->info("Index is live — documents become searchable as the queue processes.");
return $alias;
}
/**
* Prepare the staging index for a blue-green build.
*
* - If a staging build exists and --fresh is not set, resume it.
* - Otherwise create a fresh timestamped index and save it as staging.
*/
protected function prepareStagingIndex(
Client $client,
string $alias,
SettingsRepositoryInterface $settings,
iterable $seeders
): string {
$staging = $settings->get('blomstra-search.staging-index');
if ($this->option('fresh') && $staging) {
if ($client->indices()->exists(['index' => $staging])) {
$client->indices()->delete(['index' => $staging]);
$this->info("Dropped staging index: $staging");
}
$staging = null;
$settings->set('blomstra-search.staging-index', null);
}
if ($staging && $client->indices()->exists(['index' => $staging])) {
$this->info("Resuming staging index build: $staging");
return $staging;
}
$staging = $alias . '_' . date('YmdHis');
$client->indices()->create([
'index' => $staging,
'body' => ['settings' => $this->buildIndexSettings($settings)],
]);
$settings->set('blomstra-search.staging-index', $staging);
foreach ($seeders as $seeder) {
$this->setContinueAt($settings, $seeder->type(), null);
}
$this->info("Created staging index: $staging");
return $staging;
}
/**
* Persist the analysis config and compat version that are actually live in ES for the
* given index. Reading from ES (rather than from Flarum settings) means rollbacks are
* also covered: the stored values always reflect the index that is currently aliased,
* not the settings at the time the command ran. Indexes built before _meta tracking
* existed will yield a null compat version, which correctly triggers the reindex warning.
*/
protected function saveIndexedConfig(Client $client, SettingsRepositoryInterface $settings, string $indexName): void
{
/** @var SettingsRepositoryInterface $settings */
$settings = resolve(SettingsRepositoryInterface::class);
$settingsResponse = $client->indices()->getSettings(['index' => $indexName]);
$analysis = Arr::get($settingsResponse, "$indexName.settings.index.analysis", []);
$key = "blomstra-search.continued-at.$type";
$analyzer = Arr::get($analysis, 'analyzer.flarum_analyzer.type', 'english');
$stemExclusion = Arr::get($analysis, 'analyzer.flarum_analyzer.stem_exclusion', []);
if ($at) {
$settings->set($key, $at);
} else {
return $settings->get($key);
$mappingResponse = $client->indices()->getMapping(['index' => $indexName]);
$compatVersion = Arr::get($mappingResponse, "$indexName.mappings._meta.index_compat_version");
$settings->set('blomstra-search.indexed-analyzer', $analyzer);
$settings->set('blomstra-search.indexed-stem-exclusion', implode("\n", $stemExclusion));
$settings->set('blomstra-search.index-compatible', $compatVersion);
}
protected function buildIndexSettings(SettingsRepositoryInterface $settings): array
{
$language = $settings->get('blomstra-search.analyzer-language') ?: 'english';
$raw = $settings->get('blomstra-search.stem-exclusion', '');
$stemExclusion = array_values(array_filter(array_map('trim', explode("\n", $raw))));
if ($language === 'cjk') {
// CJK uses the built-in bigram analyzer; no autocomplete subfield.
return [
'analysis' => [
'analyzer' => [
'flarum_analyzer' => ['type' => 'cjk'],
],
],
];
}
$analyzerConfig = ['type' => $language];
if (!empty($stemExclusion)) {
$analyzerConfig['stem_exclusion'] = $stemExclusion;
}
return [
'index.max_ngram_diff' => self::TITLE_MAX_GRAM - self::TITLE_MIN_GRAM,
'analysis' => [
'filter' => [
'title_autocomplete_filter' => [
'type' => 'edge_ngram',
'min_gram' => self::TITLE_MIN_GRAM,
'max_gram' => self::TITLE_MAX_GRAM,
'token_chars' => ['letter', 'digit'],
],
],
'analyzer' => [
// Symmetric language analyzer: used for BOTH index and search on content + title.
'flarum_analyzer' => $analyzerConfig,
// Title autocomplete index-time: prefix n-grams, no stemming.
'flarum_title_autocomplete' => [
'type' => 'custom',
'tokenizer' => 'standard',
'filter' => ['lowercase', 'title_autocomplete_filter'],
],
// Title autocomplete search-time: just lowercase — predictable on partial words.
'flarum_title_search' => [
'type' => 'custom',
'tokenizer' => 'standard',
'filter' => ['lowercase'],
],
],
],
];
}
protected function mappingProperties(): array
{
$language = resolve(SettingsRepositoryInterface::class)->get('blomstra-search.analyzer-language') ?: 'english';
$titleMapping = [
'type' => 'text',
'analyzer' => 'flarum_analyzer',
];
// Only add the autocomplete subfield for non-CJK languages.
if ($language !== 'cjk') {
$titleMapping['fields'] = [
'autocomplete' => [
'type' => 'text',
'analyzer' => 'flarum_title_autocomplete',
'search_analyzer' => 'flarum_title_search',
],
];
}
$contentMapping = [
'type' => 'text',
'analyzer' => 'flarum_analyzer',
];
// Only add the autocomplete subfield for non-CJK languages.
if ($language !== 'cjk') {
$contentMapping['fields'] = [
'autocomplete' => [
'type' => 'text',
'analyzer' => 'flarum_title_autocomplete',
'search_analyzer' => 'flarum_title_search',
],
];
}
return [
'_meta' => ['index_compat_version' => self::INDEX_COMPAT_VERSION],
'properties' => [
'join_field' => ['type' => 'join', 'relations' => ['discussion' => 'post']],
'discussion_id' => ['type' => 'integer'],
'content' => $contentMapping,
'title' => $titleMapping,
'rawId' => ['type' => 'integer'],
'created_at' => ['type' => 'date'],
'updated_at' => ['type' => 'date'],
'is_private' => ['type' => 'boolean'],
'user_id' => ['type' => 'integer'],
'groups' => ['type' => 'keyword'],
'comment_count' => ['type' => 'integer'],
'tags' => ['type' => 'keyword'],
'is_sticky' => ['type' => 'boolean'],
'is_hidden' => ['type' => 'boolean'],
],
];
}
/**
* Query ES for rawIds already indexed in $targetIndex for the given $joinRelation and ID range.
* Retries on transient ES failures (NoNodesAvailableException / other ElasticsearchException)
* by sleeping past the StaticNoPingConnectionPool dead-node timeout (default 60 s) before
* each retry, giving the pool a chance to resurface the node.
*/
protected function queryIndexedIds(
Client $client,
string $targetIndex,
string $joinRelation,
int $rangeFrom,
int $rangeTo,
int $maxRetries = 10
): array {
$attempt = 0;
while (true) {
try {
$response = (new Builder($client))
->index($targetIndex)
->size(2500)
->addQuery(
(new BoolQuery())
->add((new RangeQuery('rawId'))->gte($rangeFrom)->lte($rangeTo))
->add(TermQuery::create('join_field', $joinRelation))
)
->search();
return Arr::pluck(Arr::get($response, 'hits.hits'), '_source.rawId');
} catch (ElasticsearchException $e) {
$attempt++;
if ($attempt >= $maxRetries) {
throw $e;
}
$this->warn("ES error on range {$rangeFrom}{$rangeTo} (attempt {$attempt}/{$maxRetries}): {$e->getMessage()}. Waiting 65 s before retry…");
sleep(65); // outlast the StaticNoPingConnectionPool dead-node window (default 60 s)
}
}
}
protected function getContinueAt(SettingsRepositoryInterface $settings, string $type): ?int
{
$raw = $settings->get("blomstra-search.continued-at.$type");
return $raw !== null ? (int) $raw : null;
}
protected function setContinueAt(SettingsRepositoryInterface $settings, string $type, ?int $at): void
{
$settings->set("blomstra-search.continued-at.$type", $at);
}
}

View File

@ -0,0 +1,41 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Elasticsearch;
class BoolQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery
{
protected ?int $minimumShouldMatch = null;
public static function create(): static
{
return new self();
}
public function minimumShouldMatch(int $minimum): static
{
$this->minimumShouldMatch = $minimum;
return $this;
}
public function toArray(): array
{
$array = parent::toArray();
if ($this->minimumShouldMatch !== null) {
$array['bool']['minimum_should_match'] = $this->minimumShouldMatch;
}
return $array;
}
}

View File

@ -0,0 +1,57 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
class HasChildQuery implements Query
{
protected bool $innerHits = false;
public function __construct(
protected string $type,
protected Query $query,
protected string $scoreMode = 'sum'
) {}
public static function create(string $type, Query $query, string $scoreMode = 'sum'): static
{
return new static($type, $query, $scoreMode);
}
/**
* Include inner_hits so the best-matching post ID is available in the response.
* ES returns the top-scoring child document per parent hit under inner_hits.best_post.
*/
public function withInnerHits(): static
{
$this->innerHits = true;
return $this;
}
public function toArray(): array
{
$hasChild = [
'type' => $this->type,
'score_mode' => $this->scoreMode,
'query' => $this->query->toArray(),
];
if ($this->innerHits) {
$hasChild['inner_hits'] = ['name' => 'best_post', 'size' => 1];
}
return ['has_child' => $hasChild];
}
}

View File

@ -1,6 +1,16 @@
<?php
namespace Blomstra\Search\Elasticsearch;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery;
@ -22,7 +32,7 @@ class MatchPhraseQuery extends MatchQuery
$query[$this->field]['boost'] = $this->boost;
return [
'match_phrase' => $query
'match_phrase' => $query,
];
}
}

View File

@ -1,6 +1,16 @@
<?php
namespace Blomstra\Search\Elasticsearch;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Elasticsearch;
class MatchQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery
{
@ -52,6 +62,10 @@ class MatchQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery
$query['match'][$this->field]['boost'] = $this->boost;
$query['match'][$this->field]['zero_terms_query'] = $this->zeroTerms ? 'all' : 'none';
if ($this->analyzer !== null) {
$query['match'][$this->field]['analyzer'] = $this->analyzer;
}
return $query;
}
}

View File

@ -1,6 +1,16 @@
<?php
namespace Blomstra\Search\Elasticsearch;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
@ -42,8 +52,8 @@ class SimpleSearchQuery implements Query
'fields' => $this->fields,
'analyzer' => $this->analyzer,
'default_operator' => 'AND',
'boost' => $this->boost
]
'boost' => $this->boost,
],
];
}
}

View File

@ -1,6 +1,16 @@
<?php
namespace Blomstra\Search\Elasticsearch;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;

View File

@ -1,6 +1,16 @@
<?php
namespace Blomstra\Search\Elasticsearch;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Elasticsearch;
class WildcardQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\WildcardQuery
{
@ -34,11 +44,10 @@ class WildcardQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\WildcardQu
$query = parent::toArray();
$query['wildcard'][$this->field]['boost'] = $this->boost;
$query['wildcard'][$this->field]['case_insensitive'] = ! $this->sensitivity;
$query['wildcard'][$this->field]['case_insensitive'] = !$this->sensitivity;
if ($this->rewrite) {
$query['wildcard'][$this->field]['rewrite'] = $this->rewrite;
}
return $query;

View File

@ -1,13 +1,22 @@
<?php
namespace Blomstra\Search\Exceptions;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Exceptions;
use Throwable;
class SeedingException extends \Exception
{
public function __construct($message = "", public array $items, $code = 0, Throwable $previous = null)
public function __construct($message = '', public array $items, $code = 0, Throwable $previous = null)
{
parent::__construct($message, $code, $previous);
}

View File

@ -1,6 +1,16 @@
<?php
namespace Blomstra\Search\Jobs;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Jobs;
use Elasticsearch\Client;
use Illuminate\Database\Eloquent\Model;
@ -9,21 +19,24 @@ class DeletingJob extends Job
{
public function handle(Client $client)
{
if ($this->models->isEmpty()) return;
if ($this->models->isEmpty()) {
return;
}
// Preparing body for storing.
$body = $this->models->map(function (Model $model) {
$document = $this->seeder->toDocument($model);
$routing = $this->seeder->routing($model);
return [
['delete' => ['_index' => $this->index, '_id' => $document->id]]
['delete' => ['_index' => $this->index, '_id' => $document->id, 'routing' => $routing]],
];
})->flatten(1);
$response = $client->bulk([
'index' => $this->index,
'body' => $body->toArray(),
'refresh' => true
'refresh' => true,
]);
}
}

View File

@ -1,10 +1,20 @@
<?php
namespace Blomstra\Search\Jobs;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
use Blomstra\Search\Seeders\Seeder;
namespace Sbp\Search\Jobs;
use Sbp\Search\Seeders\Seeder;
use Flarum\Queue\AbstractJob;
use Illuminate\Support\Collection;
use Illuminate\Database\Eloquent\Collection;
abstract class Job extends AbstractJob
{
@ -12,10 +22,16 @@ abstract class Job extends AbstractJob
public static ?string $onQueue = null;
public function __construct(protected Collection $models, protected Seeder $seeder)
/**
* @param string|null $targetIndex Explicit index name for blue-green builds.
* Defaults to the configured alias when null.
*/
public function __construct(protected Collection $models, protected Seeder $seeder, ?string $targetIndex = null)
{
$this->index = resolve('blomstra.search.elastic_index');
$this->index = $targetIndex ?? resolve('blomstra.search.elastic_index');
if (static::$onQueue) $this->onQueue(static::$onQueue);
if (static::$onQueue) {
$this->onQueue(static::$onQueue);
}
}
}

View File

@ -1,44 +0,0 @@
<?php
namespace Blomstra\Search\Jobs;
use Blomstra\Search\Exceptions\SeedingException;
use Elasticsearch\Client;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Support\Arr;
class SavingJob extends Job
{
public function handle(Client $client)
{
if ($this->models->isEmpty()) return;
// Preparing body for storing.
$body = $this->models->map(function (Model $model) {
$document = $this->seeder->toDocument($model);
return [
['index' => ['_index' => $this->index, '_id' => $document->id]],
$document->toArray()
];
})
->flatten(1);
$response = $client->bulk([
'index' => $this->index,
'body' => $body->toArray(),
'refresh' => true
]);
if (Arr::get($response, 'errors') !== true) return true;
$items = Arr::get($response, 'items');
$error = Arr::get(Arr::first($items), 'index.error.reason');
throw new SeedingException(
"Failed to seed: $error",
$items
);
}
}

View File

@ -0,0 +1,62 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Jobs;
use Sbp\Search\Exceptions\SeedingException;
use Elasticsearch\Client;
use Illuminate\Database\Eloquent\Collection as EloquentCollection;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Support\Arr;
class UpdateSearchJob extends Job
{
public function handle(Client $client)
{
if ($this->models->isEmpty()) {
return;
}
$this->models->loadMissing($this->seeder->relationships());
// Preparing body for storing.
$body = $this->models->map(function (Model $model) {
$document = $this->seeder->toDocument($model);
$routing = $this->seeder->routing($model);
return [
['index' => ['_index' => $this->index, '_id' => $document->id, 'routing' => $routing]],
$document->toArray(),
];
})
->flatten(1);
$response = $client->bulk([
'index' => $this->index,
'body' => $body->toArray(),
]);
if (Arr::get($response, 'errors') !== true) {
return true;
}
$items = Arr::get($response, 'items');
$failed = array_filter($items, fn ($item) => isset($item['index']['error']));
$error = Arr::get(Arr::first($failed), 'index.error.reason', 'unknown error');
throw new SeedingException(
"Failed to seed: $error (" . count($failed) . '/' . count($items) . ' items failed)',
$items
);
}
}

View File

@ -0,0 +1,59 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Jobs;
use Elasticsearch\Client;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Discussion\Discussion;
use Flarum\Queue\AbstractJob;
class ViewsSearchJob extends AbstractJob
{
protected string $index;
protected string $documentType;
public function __construct(protected int $discussionId)
{
$this->index = resolve('blomstra.search.elastic_index');
$this->documentType = resolve(DiscussionSerializer::class)->getType(new Discussion());
if (Job::$onQueue) {
$this->onQueue(Job::$onQueue);
}
}
public function handle(Client $client): void
{
$discussion = Discussion::find($this->discussionId);
if (!$discussion) {
return;
}
$type = $this->documentType;
try {
$client->update([
'index' => $this->index,
'id' => "$type:{$this->discussionId}",
'routing' => (string) $this->discussionId,
'retry_on_conflict' => 3,
'body' => [
'doc' => ['view_count' => (int) $discussion->view_count],
],
]);
} catch (\Elasticsearch\Common\Exceptions\Missing404Exception $e) {
// Document not yet indexed; will be picked up on next: blomstra:search:index fill
}
}
}

View File

@ -1,11 +1,21 @@
<?php
namespace Blomstra\Search;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
use Blomstra\Search\Jobs\DeletingJob;
use Blomstra\Search\Jobs\Job;
use Blomstra\Search\Jobs\SavingJob;
use Blomstra\Search\Seeders;
namespace Sbp\Search;
use Sbp\Search\Jobs\DeletingJob;
use Sbp\Search\Jobs\Job;
use Sbp\Search\Jobs\UpdateSearchJob;
use Sbp\Search\Jobs\ViewsSearchJob;
use Elasticsearch\Client as Elastic;
use Elasticsearch\ClientBuilder;
use Flarum\Api\Client;
@ -16,7 +26,7 @@ use Flarum\Settings\SettingsRepositoryInterface;
use Illuminate\Contracts\Container\Container;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Contracts\Queue\Queue;
use Illuminate\Support\Collection;
use Illuminate\Database\Eloquent\Collection;
use Laminas\Stratigility\MiddlewarePipe;
use Psr\Log\LoggerInterface;
@ -37,7 +47,13 @@ class Provider extends AbstractServiceProvider
$this->container->singleton(Elastic::class, function (Container $container) use ($settings, $config) {
$builder = ClientBuilder::create()
->setHosts([$settings->get('blomstra-search.elastic-endpoint')]);
->setHosts([$settings->get('blomstra-search.elastic-endpoint')])
->setConnectionParams([
'client' => [
'connect_timeout' => 2, // fail fast if ES is unreachable
'timeout' => 10, // allow time for complex queries
],
]);
if ($config->inDebugMode()) {
$builder->setLogger($container->make(LoggerInterface::class));
@ -53,7 +69,6 @@ class Provider extends AbstractServiceProvider
return $builder->build();
});
$this->container->instance(
'blomstra.search.elastic_index',
$settings->get('blomstra-search.elastic-index', 'flarum')
@ -62,23 +77,28 @@ class Provider extends AbstractServiceProvider
$this->container->extend(
Client::class,
function () {
$pipe = new MiddlewarePipe;
$pipe = new MiddlewarePipe();
$exclude = resolve('flarum.api_client.exclude_middleware');
$middlewareStack = array_filter(resolve('flarum.api.middleware'), function ($middlewareClass) use ($exclude) {
return ! in_array($middlewareClass, $exclude);
return !in_array($middlewareClass, $exclude);
});
foreach ($middlewareStack as $middleware) {
$pipe->pipe(resolve($middleware));
}
$pipe->pipe(new ExecuteRoute);
$pipe->pipe(new ExecuteRoute());
return new Api\Client($pipe);
}
);
$this->container->tag([
Searchers\DiscussionSearcher::class,
Searchers\CommentPostSearcher::class,
], 'blomstra.search.searchers');
}
public function boot()
@ -95,12 +115,16 @@ class Provider extends AbstractServiceProvider
/** @var string|Seeders\Seeder $seeder */
foreach ($seeders as $seeder) {
$seeder::savingOn($events, function ($model) use ($queue, $seeder) {
$queue->pushOn(Job::$onQueue, new SavingJob(Collection::make([$model]), $seeder));
$queue->pushOn(Job::$onQueue, new UpdateSearchJob(Collection::make([$model]), $seeder));
});
$seeder::deletingOn($events, function ($model) use ($queue, $seeder) {
$queue->pushOn(Job::$onQueue, new DeletingJob(Collection::make([$model]), $seeder));
});
$seeder::viewingOn($events, function (int $discussionId) use ($queue) {
$queue->pushOn(Job::$onQueue, new ViewsSearchJob($discussionId));
});
}
}
}

View File

@ -1,13 +1,24 @@
<?php
namespace Blomstra\Search\Save;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Save;
use Carbon\Carbon;
use Illuminate\Support\Arr;
use Illuminate\Support\Fluent;
/**
* @property string $type
* @property string $id
* @property int $rawId
* @property string $content
* @property Carbon $created_at
* @property Carbon $updated_at
@ -20,4 +31,13 @@ use Illuminate\Support\Fluent;
*/
class Document extends Fluent
{
/**
* Exclude `id` from the body sent to Elasticsearch. The document ID is
* already stored as `_id` by the bulk API action; duplicating it in
* `_source` wastes space and is redundant.
*/
public function toArray(): array
{
return Arr::except($this->attributes, ['id']);
}
}

View File

@ -0,0 +1,27 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Searchers;
use Sbp\Search\Seeders\CommentSeeder;
class CommentPostSearcher extends Searcher
{
protected string|null $seeder = CommentSeeder::class;
public function enabled(): bool
{
$enabled = $this->setting('blomstra-search.search-post-bodies', true);
return boolval($enabled);
}
}

View File

@ -0,0 +1,32 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Searchers;
use Sbp\Search\Seeders\DiscussionSeeder;
class DiscussionSearcher extends Searcher
{
protected string|null $seeder = DiscussionSeeder::class;
public function enabled(): bool
{
$enabled = $this->setting('blomstra-search.search-discussion-subjects', true);
return boolval($enabled);
}
public function boost(): float
{
return 1.5;
}
}

View File

@ -0,0 +1,48 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Searchers;
use Sbp\Search\Seeders\Seeder;
use Flarum\Settings\SettingsRepositoryInterface;
abstract class Searcher
{
protected string|null $seeder = null;
public function type(): string
{
/** @var Seeder $seeder */
$seeder = $this->seeder;
if (empty($seeder)) {
throw new \InvalidArgumentException('Implement type or add $seeder');
}
return (new $seeder())->type();
}
public function enabled(): bool
{
return true;
}
public function boost(): float
{
return 1;
}
protected function setting(string $key, $default = null)
{
return resolve(SettingsRepositoryInterface::class)->get($key, $default);
}
}

View File

@ -1,12 +1,23 @@
<?php
namespace Blomstra\Search\Seeders;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
use Blomstra\Search\Save\Document;
namespace Sbp\Search\Seeders;
use Sbp\Search\Save\Document;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Api\Serializer\PostSerializer;
use Flarum\Discussion\Discussion;
use Flarum\Post\CommentPost;
use Flarum\Post\Event\Deleted;
use Flarum\Post\Event\Posted;
use Flarum\Post\Event as Core;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Model;
@ -15,66 +26,71 @@ class CommentSeeder extends Seeder
{
public function type(): string
{
return resolve(PostSerializer::class)->getType(new CommentPost);
return resolve(PostSerializer::class)->getType(new CommentPost());
}
public function joinRelation(): string
{
return 'post';
}
public function routing(Model $model): string
{
return (string) $model->discussion_id;
}
public function query(): Builder
{
$includes = ['discussion'];
if ($this->extensionEnabled('flarum-tags')) {
$includes[] = 'discussion.tags';
}
if ($this->extensionEnabled('fof-byobu')) {
$includes[] = 'discussion.recipientUsers';
$includes[] = 'discussion.recipientGroups';
}
return CommentPost::query()
->whereNull('hidden_at')
->where('type', CommentPost::$type)
->with($includes);
->where('type', CommentPost::$type);
}
public static function savingOn(Dispatcher $events, callable $callable)
{
$events->listen(Posted::class, function (Posted $event) use ($callable) {
$events->listen([
Core\Posted::class,
Core\Revised::class,
Core\Hidden::class,
Core\Restored::class,
], function ($event) use ($callable) {
$callable($event->post);
});
}
public static function deletingOn(Dispatcher $events, callable $callable)
{
$events->listen(Deleted::class, function (Deleted $event) use ($callable) {
$events->listen([
Core\Deleted::class
], function ($event) use ($callable) {
$callable($event->post);
});
}
/** Cached discussion type string (e.g. "discussions") — resolved once per job. */
private ?string $discussionType = null;
private function discussionType(): string
{
return $this->discussionType ??= resolve(DiscussionSerializer::class)->getType(new Discussion());
}
/**
* Post documents only need content for has_child matching and the join
* field to establish the parent-child relationship. All discussion-level
* fields (groups, tags, comment_count, etc.) live on the discussion
* document and are irrelevant here.
*
* @param CommentPost $model
* @return Document
*/
public function toDocument(Model $model): Document
{
$document = new Document([
'type' => $this->type(),
'id' => $this->type() . ':' . $model->id,
return new Document([
'join_field' => ['name' => $this->joinRelation(), 'parent' => "{$this->discussionType()}:{$model->discussion_id}"],
'discussion_id' => $model->discussion_id,
'id' => $this->type().':'.$model->id,
'rawId' => $model->id,
'content' => $model->content,
'content_partial' => $model->content,
'created_at' => $model->created_at?->toAtomString(),
'updated_at' => $model->edited_at?->toAtomString(),
'is_private' => $model->is_private,
'user_id' => $model->user_id,
'groups' => $this->groupsForDiscussion($model->discussion),
'comment_count' => $model->discussion->comment_count,
'is_hidden' => $model->hidden_at !== null,
]);
if ($this->extensionEnabled('fof-byobu')) {
$document['recipient_users'] = $model->discussion->recipientUsers->pluck('id')->toArray();
$document['recipient_groups'] = $model->discussion->recipientGroups->pluck('id')->toArray();
}
return $document;
}
}

View File

@ -1,26 +1,56 @@
<?php
namespace Blomstra\Search\Seeders;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
use Blomstra\Search\Save\Document;
namespace Sbp\Search\Seeders;
use Sbp\Search\Save\Document;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Discussion\Discussion;
use Flarum\Discussion\Event\Deleted;
use Flarum\Discussion\Event\Hidden;
use Flarum\Discussion\Event\Restored;
use Flarum\Discussion\Event\Started;
use Flarum\Discussion\Event as Core;
use Flarum\Extension\ExtensionManager;
use Flarum\Group\Group;
use Flarum\Group\Permission;
use Flarum\Tags\Tag;
use FoF\Byobu\Events as Byobu;
use FoF\DiscussionViews\Events\DiscussionWasViewed;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder;
use Illuminate\Support\Collection;
use Illuminate\Database\Eloquent\Model;
class DiscussionSeeder extends Seeder
{
public function type(): string
{
return resolve(DiscussionSerializer::class)->getType(new Discussion);
return resolve(DiscussionSerializer::class)->getType(new Discussion());
}
public function joinRelation(): string
{
return 'discussion';
}
public function routing(Model $model): string
{
return (string) $model->id;
}
public function query(): Builder
{
return Discussion::query()
->whereNull('hidden_at');
}
public function relationships(): array
{
$includes = [];
@ -32,36 +62,66 @@ class DiscussionSeeder extends Seeder
$includes[] = 'recipientUsers';
$includes[] = 'recipientGroups';
}
return Discussion::query()
->whereNull('hidden_at')
->with($includes);
return $includes;
}
public static function savingOn(Dispatcher $events, callable $callable)
{
$events->listen([Started::class, Restored::class], function ($event) use ($callable) {
$events->listen([
// flarum/core events
Core\Started::class, Core\Restored::class, Core\Renamed::class,
// fof/byobu discussion recipients events.
Byobu\DiscussionMadePublic::class, Byobu\RemovedSelf::class, Byobu\RecipientsChanged::class,
], function ($event) use ($callable) {
return $callable($event->discussion);
});
}
public static function viewingOn(Dispatcher $events, callable $callable): void
{
if (!resolve(ExtensionManager::class)->isEnabled('fof-discussion-views')) {
return;
}
$events->listen(DiscussionWasViewed::class, function (DiscussionWasViewed $event) use ($callable) {
$viewCount = $event->discussion->view_count;
$shouldSync = match (true) {
$viewCount < 15 => true,
$viewCount < 100 => rand(1, 3) === 1,
default => rand(1, 19) === 1,
};
if ($shouldSync) {
$callable($event->discussion->id);
}
});
}
public static function deletingOn(Dispatcher $events, callable $callable)
{
$events->listen([Deleted::class, Hidden::class], function ($event) use ($callable) {
$events->listen([
// flarum/core events.
Core\Deleted::class, Core\Hidden::class
], function ($event) use ($callable) {
return $callable($event->discussion);
});
}
/**
* @param Discussion $model
*
* @return Document
*/
public function toDocument(Model $model): Document
{
$document = new Document([
'type' => $this->type(),
'id' => $this->type() . ':' . $model->id,
'join_field' => $this->joinRelation(),
'id' => $this->type().':'.$model->id,
'rawId' => $model->id,
'content' => $model->title,
'content_partial' => $model->title,
'title' => $model->title,
'created_at' => $model->created_at?->toAtomString(),
'updated_at' => $model->last_posted_at?->toAtomString(),
'is_private' => $model->is_private,
@ -70,15 +130,81 @@ class DiscussionSeeder extends Seeder
'comment_count' => $model->comment_count,
]);
if ($this->extensionEnabled('flarum-tags')) {
$document['tags'] = $model->tags->pluck('id')->toArray();
}
if ($this->extensionEnabled('fof-byobu')) {
$document['recipient_users'] = $model->recipientUsers->pluck('id')->toArray();
$document['recipient_groups'] = $model->recipientGroups->pluck('id')->toArray();
$document['recipient_users'] = $model->recipientUsers
->whereNull('removed_at')
->pluck('id')
->toArray();
$document['recipient_groups'] = $model->recipientGroups
->whereNull('removed_at')
->pluck('id')
->toArray();
}
if ($this->extensionEnabled('flarum-sticky')) {
$document['is_sticky'] = (bool) $model->is_sticky;
}
if ($this->extensionEnabled('fof-discussion-views')) {
$document['view_count'] = (int) ($model->view_count ?? 0);
}
return $document;
}
/**
* All viewForum permissions keyed by permission string, loaded once per seeder instance.
* Avoids N×2 Permission queries inside the per-document map loop.
*/
private ?Collection $cachedPermissions = null;
private ?Collection $cachedGlobalPermission = null;
private function allPermissions(): Collection
{
if ($this->cachedPermissions === null) {
$this->cachedPermissions = Permission::query()
->where(function ($q) {
$q->where('permission', 'viewForum')
->orWhere('permission', 'like', 'tag%.viewForum');
})
->get();
$this->cachedGlobalPermission = $this->cachedPermissions
->where('permission', 'viewForum')
->pluck('group_id');
}
return $this->cachedPermissions;
}
protected function groupsForDiscussion(Discussion $discussion): array
{
$allPerms = $this->allPermissions();
$permissions = collect();
if ($this->extensionEnabled('flarum-tags')) {
/** @var Collection $tags */
$tags = $discussion->tags;
$permissions = $tags->map(function (Tag $tag) use ($allPerms) {
$tagPerms = $allPerms->where('permission', "tag$tag->id.viewForum");
if ($tag->is_restricted) {
$tagPerms = $tagPerms->add(['group_id' => Group::ADMINISTRATOR_ID]);
}
return $tagPerms->pluck('group_id');
})->flatten();
}
if (!$discussion->is_private && $permissions->isEmpty()) {
$permissions = $this->cachedGlobalPermission;
}
return $permissions->toArray();
}
}

View File

@ -1,66 +1,54 @@
<?php
namespace Blomstra\Search\Seeders;
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
use Blomstra\Search\Save\Document;
use Flarum\Discussion\Discussion;
namespace Sbp\Search\Seeders;
use Sbp\Search\Save\Document;
use Flarum\Extension\ExtensionManager;
use Flarum\Group\Group;
use Flarum\Group\Permission;
use Flarum\Tags\Tag;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Database\Eloquent\Model;
abstract class Seeder
{
abstract public function type(): string;
/**
* The join relation name for this document type ('discussion' or 'post').
* Used in the join_field mapping and for filtering in seed-missing checks.
*/
abstract public function joinRelation(): string;
/**
* The ES routing key for this model. Parent and child documents for the same
* discussion must share a routing key so ES places them on the same shard.
*/
abstract public function routing(Model $model): string;
abstract public function query(): Builder;
abstract public static function savingOn(Dispatcher $events, callable $callable);
abstract public static function deletingOn(Dispatcher $events, callable $callable);
/** No-op default; override in seeders that need to react to view events. */
public static function viewingOn(Dispatcher $events, callable $callable): void {}
abstract public function toDocument(Model $model): Document;
protected function groupsForDiscussion(Discussion $discussion): array
/** Relationships to eager-load on the collection before calling toDocument(). */
public function relationships(): array
{
$permissions = collect();
$globalPermission = Permission::query()
->where('permission', 'viewForum')
->pluck('group_id');
if ($this->extensionEnabled('flarum-tags')) {
/** @var Collection $tags */
$tags = $discussion->tags;
$filters['tags'] = $tags->pluck('id')->toArray();
$tagPermissions = Permission::query()
->whereIn(
'permission',
$tags->pluck('id')->map(function (int $id) {
return "tag$id.viewForum";
})
)->get();
$permissions = $tags->map(function (Tag $tag) use ($tagPermissions) {
$permissions = $tagPermissions->where('permission', "tag$tag->id.viewForum");
if ($tag->is_restricted) {
$permissions = $permissions->add(['group_id' => Group::ADMINISTRATOR_ID]);
}
return $permissions->pluck('group_id');
})->flatten();
}
if (! $discussion->is_private && $permissions->isEmpty()) {
$permissions = $globalPermission;
}
return $permissions->toArray();
return [];
}
protected function extensionEnabled(string $extension): bool