Compare commits

..

67 Commits

Author SHA1 Message Date
mueh e4590f3ea9 Merge branch 'main' of https://gitea.muehlberger.net/sbp-jm/sbp-flarum-search 2026-04-28 15:40:42 +02:00
mueh 83c0b0e7ec Working stable release! 2026-04-28 15:40:09 +02:00
MauBiBot 3a8559ecca Rename internal API route reference from blomstra.search to sbp.search 2026-04-28 14:46:44 +02:00
MauBiBot 082892f413 Fix forum search API endpoint to /sbp/search 2026-04-28 11:51:34 +02:00
MauBiBot 0b9586aee2 Fix forum search API endpoint to /sbp/search 2026-04-28 11:51:15 +02:00
mueh c54d916361 richitger api pfad 2026-04-28 11:49:57 +02:00
jm 09c01b3445 glaub nicht dass das böse war 2026-04-28 09:30:40 +00:00
jm a089698247 register provider evtl. böse? 2026-04-28 09:25:30 +00:00
mueh 7344fc33b7 revert to old extend.php 2026-04-28 11:06:47 +02:00
jm 2696fa4dfd wieder ohne extensionid 2026-04-28 09:00:51 +00:00
mueh a6d6230553 typo 2026-04-28 10:57:15 +02:00
mueh f537506166 Merge branch 'main' of https://gitea.muehlberger.net/sbp-jm/sbp-flarum-search 2026-04-28 10:49:51 +02:00
mueh a9bb20dfe7 extension id angegeben 2026-04-28 10:49:22 +02:00
MauBiBot 6417e8e9f4 Bind admin settings to sbp-search extension page 2026-04-28 10:31:54 +02:00
mueh 22e2796a65 admin js hatte noch alte blomstra-search drin 2026-04-28 10:27:28 +02:00
mueh 52168bb5c3 js wieder gefixt aus anderem commit 2026-04-28 10:01:50 +02:00
mueh 7f4befd73b composer fix 2026-04-28 09:37:39 +02:00
mueh 115c06e9bf Merge branch 'sbp-hardfork' 2026-04-28 09:07:16 +02:00
jm d2b8869eb4 composer update 2026-04-22 22:01:57 +00:00
MauBiBot f72acc60ef Rename blomstra/search fork to sbp/flarum-search (Phase 1: namespaces, routes, JS initializers) 2026-04-22 22:23:47 +02:00
JM fc830ea921 Add content autocomplete to search dropdown 2026-04-22 20:52:35 +02:00
MauBiBot 4538fe2ba8 Add prefix search for content and enable autocomplete 2026-04-22 20:43:33 +02:00
JM 5774879331 fuzzy search in thread 2026-04-22 20:19:09 +02:00
MauBiBot 3e2f3f9585 always use ES search on initial search page load 2026-04-22 19:46:09 +02:00
MauBiBot bf90e3279e remove frontend default sort from discussion search state 2026-04-22 19:31:30 +02:00
MauBiBot bce3258325 fix forum search route and rebuild compiled assets 2026-04-22 19:05:46 +02:00
MauBiBot 7f282cdb84 merge local search fixes after pull 2026-04-22 18:56:43 +02:00
JM 7cb4071a54 restore strong title match and keep prefix title fallback 2026-04-22 18:08:12 +02:00
JM e3dfc04192 add low-boost title autocomplete match to full search 2026-04-22 17:50:30 +02:00
MauBiBot e7e4495306 rebuild compiled JS assets for analyzer selector 2026-04-22 16:36:24 +02:00
mueh 0fa8fb92eb ai forgot to commit 2026-04-22 15:47:54 +02:00
mueh 925ecb0e7c tune title autocomplete edge_ngram bounds 2026-04-22 12:33:13 +02:00
Bart van Bragt e92307cf16 build: commit compiled JS dist files for feat/analyzer-refactor branch 2026-04-16 22:07:28 +02:00
Bart van Bragt d27d71ff72 style: apply prettier formatting to admin/index.ts 2026-04-16 11:57:59 +02:00
Bart van Bragt f7d9935fb9 feat: suppress ES instant results when search-post-bodies is enabled
When searching inside comments is ON, the autocomplete dropdown now returns
nothing from ES (only usernames show). This avoids showing title-only matches
that are inconsistent with the full comment-body search results.

Also passes filter[autocomplete]=1 to the API so the server uses the fast
title.autocomplete path instead of the full has_child query.
2026-04-16 11:57:34 +02:00
Bart van Bragt 33d4946947 feat: stem exclusion admin setting and analyzer language selector improvements
Add a stem-exclusion textarea to the admin settings page. Words entered here
(one per line) are passed as stem_exclusion to the language analyzer, preventing
them from being stemmed. Requires a full re-index to take effect.

The two language-dependent fields (stem exclusion, min search length) are
disabled when CJK is selected, since neither applies to bigram analysis.

Settings are reordered: search-scope switches now appear above the language
block so the language-dependent fields sit directly below the language selector.

Replace the lowercase ES identifier labels in the language dropdown with proper
display names (e.g. "Brazilian Portuguese", "CJK (Chinese, Japanese, Korean)",
"Sorani (Kurdish)"). Drift detection now tracks stem exclusion instead of the
removed min_gram index setting.
2026-04-16 11:52:34 +02:00
Bart van Bragt 0f58abf141 feat: add title.autocomplete instant search path sorted by latest
When filter[autocomplete]=1 is present, bypass the full has_child query
and run a single MatchQuery on title.autocomplete (edge-ngram prefix match).
Results are sorted by updated_at desc to stay consistent with full search.
2026-04-16 11:51:45 +02:00
Bart van Bragt 0adaab5d84 fix: resolve analyzer mismatch with symmetric language analyzer and title field
Replace the asymmetric ngram (index) / language (search) setup on content
with a symmetric language analyzer at both index and search time, eliminating
zero-hit cases caused by stemmed tokens falling below min_gram (e.g. Turkish).

Add a title field to discussion documents with a title.autocomplete edge-ngram
subfield (min=2, max=15) for prefix instant search, and a flarum_title_search
analyzer (lowercase only) as its search-time counterpart.

Remove flarum_analyzer_partial / partial_search_filter / search_analyzer from
the content mapping — no longer needed with symmetric analysis.

Bump INDEX_COMPAT_VERSION to v3. Also fix MatchQuery::toArray() to serialize
the analyzer property which was stored but never emitted.
2026-04-16 11:51:28 +02:00
Bart van Bragt c55db0abd6 fix: show TerminalPost in search results when field sort is active or mostRelevantPost is unavailable
When params.q is set, Flarum core suppresses TerminalPost and only shows an
excerpt from mostRelevantPost. This leaves the info section empty when the post
is null (e.g. hidden/deleted first post in production) or when results are
ordered by a field sort where an excerpt is meaningless.

Extend DiscussionListItem.infoItems to:
- Replace excerpt with TerminalPost when a field sort (latest, oldest, top) is
  active — relevance excerpts are not meaningful when sorted by date/count.
- Fall back to TerminalPost when no excerpt was added (mostRelevantPost null).
2026-04-15 21:24:55 +02:00
Bart van Bragt c5643bed31 fix: default search sort to latest and keep dropdown label in sync
- Search with no explicit sort now defaults to latest (updated_at desc)
  on both backend and frontend, matching the forum UX expectation.
- "All discussions" link from the search box routes to /?q=...&sort=latest
  so the sort dropdown shows "Latest" immediately after submission.
- extendDiscussionState injects sort=-lastPostedAt into API params when
  no sort is present, keeping direct-URL navigation (?q=anan) consistent
  with the backend default.
- Backend guard changed from phpSortField===null to empty($sorts), which
  is the correct sentinel since $sorts is the authoritative ES sort list.
2026-04-15 17:21:10 +02:00
Bart van Bragt 6bcd2becd5 fix: report actual failed item error in bulk seeding exception 2026-04-15 14:08:04 +02:00
Bart van Bragt 2706ed6be0 fix: index discussion on rename 2026-04-15 14:07:55 +02:00
Bart van Bragt 557604e426 fix: retry fill ES queries on transient connection failure 2026-04-15 14:07:43 +02:00
Bart van Bragt 08353d2473 perf: sort by date by default, skip scoring on field sorts
- Default sort changed from relevance to updated_at desc. This lets
  has_child use score_mode:none, which skips child scoring entirely
  and lets ES short-circuit early on large corpora.
- When a field sort is requested, score_mode is also none for the same
  reason. score_mode:sum is only used when sorting by relevance.
- track_total_hits:false avoids a full-index count on every query,
  allowing ES to stop once it has collected enough results.
- Strip all gambit operators (tag:foo, author:bar, is:unread, etc.)
  from the ES query string, not just is:private. Leaving them in caused
  operator:and to require the gambit tokens to appear literally in post
  content, producing zero results when gambits were combined with text.
2026-04-14 22:40:43 +02:00
Bart van Bragt 80357e6a77 fix: include --staging in build conflict suggestions 2026-04-14 16:46:06 +02:00
Bart van Bragt d8eebb183a fix: prevent ES CPU saturation from has_child scoring all posts
Without minimum_should_match=1 on the inner post bool query, Elasticsearch
defaults MSM to 0 whenever a filter clause is present. This caused has_child
to score every non-hidden post on every search request, saturating CPU on
shared ES nodes with large corpora (3.7M+ posts).

- Add BoolQuery subclass with create() override and minimumShouldMatch()
  (spatie/elasticsearch-query-builder 1.x uses `new self()` in create(),
  so subclassing requires overriding it)
- Set minimumShouldMatch(1) on the inner post query after adding the
  is_hidden filter, so only genuinely matching posts are scored
- Remove the operator('or') clause from buildShouldClauses() — with
  Turkish min_ngram=2, 'or' generates 2-gram tokens that match nearly
  every post, causing near-total index scans
- Add ES client timeouts (connect: 2s, query: 10s) to prevent Apache
  mod_php worker saturation when ES is slow or unreachable
2026-04-14 16:07:50 +02:00
Bart van Bragt c54f37349f fix: use Support\Collection for cachedGlobalPermission — pluck() returns plain Collection, not Eloquent 2026-04-14 11:27:02 +02:00
Bart van Bragt ca8c248870 fix: use Eloquent Collection in job — loadMissing does not exist on Support\Collection 2026-04-14 09:59:02 +02:00
Bart van Bragt bf37e01eb0 fix: read forum attribute lazily inside extend callback 2026-04-14 09:33:17 +02:00
Bart van Bragt 4709b9d1db fix: is_hidden boolean coercion and forum settings access 2026-04-14 08:59:19 +02:00
flarum-bot bf19eed20f Bundled output for commit 2fc66b9bf7
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2026-04-13 20:22:53 +00:00
Bart van Bragt 2fc66b9bf7 perf: eager-load relationships in UpdateSearchJob and cache permissions in DiscussionSeeder
- Add relationships() to Seeder base class; DiscussionSeeder implements it to
  return tags/recipientUsers/recipientGroups for eager loading
- UpdateSearchJob calls loadMissing(relationships()) before the map loop,
  eliminating N×R lazy-load queries per batch
- Remove refresh:true from bulk call — synchronous ES refresh was the primary
  throughput bottleneck during indexing
- Cache viewForum permissions in DiscussionSeeder per job instance, replacing
  N×2 Permission queries per document with a single cached query
- Increase seeder batch size from 1000 to 2500
2026-04-13 22:18:14 +02:00
Bart van Bragt 439c74f0a3 fix: store index compat version in ES mapping _meta for correct rollback behaviour
- Write INDEX_COMPAT_VERSION into mapping _meta.index_compat_version on every
  build/promote so the version travels with the index rather than being a
  separate settings value
- saveIndexedConfig now reads compat version back from _meta alongside the
  existing analyzer/min_gram sync, so rollback automatically restores the
  correct value (or null for pre-feature indexes, triggering the reindex warning)
- Replace InvalidArgumentException in buildIndexSettings with $this->error()
  fallback to avoid stack traces in the console
- Fix stale comment in ViewsSearchJob referencing removed CLI flags
2026-04-10 13:46:08 +02:00
Bart van Bragt f486292f15 feat: make ngram min_gram configurable via admin setting
Adds a 'Minimum search query length' setting (1–4, default 3). The admin
shows a warning when the analyzer or min-search-length have changed since
the last index build.
2026-04-10 13:19:22 +02:00
Bart van Bragt 4412583cd9 chore: fix prettier formatting and npm audit vulnerabilities
Reformat admin/index.ts to satisfy prettier checks. Update package-lock.json
via npm audit fix to resolve 17 vulnerabilities in transitive dev dependencies
(webpack, babel, lodash, ws, semver, etc.).
2026-04-10 11:47:38 +02:00
Bart van Bragt 81e7b6502d feat: Show admin dashboard notice when search index needs rebuilding
After a mapping change (e.g. the parent-child restructure), existing installs
need to rebuild their index. A DashboardWidget appears on the admin page
warning the admin and showing the command to run. Compatibility is tracked
via the blomstra-search.index-compatible setting.
2026-04-10 11:10:53 +02:00
Bart van Bragt 3039b65946 refactor: Simplify index command with better naming/options
The flag-based interface (--rebuild etc) became rather convoluted with the changes. Replace this with commands with more straight forward naming.

Also added an explicit 'rollback' action and updated the README.md
2026-04-10 09:35:49 +02:00
Bart van Bragt e2e2d1428f feat: restructure index as parent-child documents with blue-green rebuilds
- Switch to ES parent/child join_field: discussion docs hold all metadata,
  post docs hold content only. Routing ensures parent and child land on the
  same shard. Removes the old flat-document approach where post data was
  duplicated onto every comment document.

- Add HasChildQuery with inner_hits so the best-matching post ID is
  surfaced as mostRelevantPost without a second DB query.

- Blue-green index rebuilds: --recreate writes into a timestamped pending
  index, --swap atomically promotes it via alias. Interrupted builds are
  resumable with --recreate --continue. --swap requires confirmation before
  proceeding.

- Index hidden posts; non-moderators are filtered at query time via
  is_hidden on the has_child clause. Core\Hidden and Core\Restored trigger
  re-indexing so moderators with post.hide can search hidden posts.
2026-04-09 23:30:57 +02:00
Bart van Bragt 80d66c98c4 feat: index and sync view_count for fof/discussion-views sort support
- Rename SavingJob to UpdateSearchJob
- Add ViewsSearchJob for lightweight partial view_count updates
- Add probabilistic sync throttle in DiscussionSeeder::viewingOn()
  to avoid an ES update on every page view
- Add view_count to mapping, DiscussionSeeder::toDocument(), and
  translateSort
- Log a warning and skip instead of 500ing on unknown sort fields
2026-04-09 12:05:28 +02:00
Bart van Bragt ff2a252e99 fix: editing a post does not update the search index
CommentSeeder only listened to Post\Event\Posted, so post edits
left the Elasticsearch document stale.
2026-04-09 11:17:17 +02:00
Bart van Bragt 5929a191f6 fix: --seed-missing stops early when a full range is already indexed
When all documents in a range were already in the index, the empty
collection caused continueAt to be set to null, aborting before
scanning lower ID ranges. Now advances past the range bottom instead.
2026-04-09 11:17:17 +02:00
Bart van Bragt 6699c8cb58 fix: ngram min_gram=1 indexes tokens that can never be matched
Flarum's search UI enforces a minimum query length of 3 characters
(MIN_SEARCH_LEN in Search.tsx), so 1- and 2-character ngram tokens
are indexed but can never be queried. Changed min_gram from 1 to 3
and updated max_ngram_diff accordingly.
2026-04-09 11:04:27 +02:00
Bart van Bragt 369ee504f3 fix: remove unused content_partial field from seeders
content_partial was introduced alongside a partialMatch() query that
targeted it. That query was later switched to use the content field
instead, but content_partial was never removed from the seeders,
causing every document to store and index its content twice.
2026-04-09 08:39:58 +02:00
Bart van Bragt eeb091273d fix: tag filter produces empty results when browsing a tag
When navigating to /t/{tag}?q=query the tag filter from the request
was silently ignored. Added a TermsQuery on the tags field using the
tag IDs resolved from the provided slugs.
2026-04-09 08:20:42 +02:00
Daniël Klabbers d252a17586 fix: not changing recipients visibility when recipients change 2024-06-13 19:27:14 +02:00
Daniël Klabbers a9dc8bfb16 fix: byobu discussions open up while using this extension 2024-06-12 11:34:18 +02:00
StyleCI Bot 82e6ec786f
Apply fixes from StyleCI 2023-10-18 13:57:41 +00:00
53 changed files with 3418 additions and 2188 deletions

1
.gitignore vendored
View File

@ -2,3 +2,4 @@ js/node_modules
js/dist
vendor/
composer.lock
analysis/*

166
README.md
View File

@ -5,47 +5,177 @@ fulltext search with one that is completely relying on the proven elasticsearch
## Features
- Sync discussions to elastic search using your queue, unobtrusively for the user.
- Reduces search loading times to well below 400ms (local tests with 50.000 discussion **260ms**)
- Sync discussions and posts to Elasticsearch using your queue, unobtrusively for the user.
- Reduces search loading times to well below 400ms (local tests with 50,000 discussions: **260ms**)
- Uses Flarum's group permissions and tags system.
- Compatible with Friends of Flarum Byōbu.
## Requirements
- Elasticsearch 7.x or OpenSearch 1.x+
- A non-sync queue driver with a running worker (`php flarum queue:work`) is strongly recommended for production. The extension works with the default sync driver, but index jobs run inline which adds latency to user-facing changes like posting.
## Installation
Use composer:
```bash
composer require sbp/flarum-search
```
## Notes
This fork is now maintained as `sbp/flarum-search`.
Intentional compatibility leftovers:
- settings keys remain `blomstra-search.*`
- some translation keys remain `blomstra-search.*`
- CLI command names may still remain `blomstra:*` until a later migration
Search behavior:
- title autocomplete enabled
- fuzzy title search enabled
- exact + fuzzy content search on full search page
- no content autocomplete in dropdown (performance decision)
``
Repository can be found here: https://gitea.muehlberger.net/sbp-jm/sbp-flarum-search
Enable the extension in the admin area and configure the Elasticsearch endpoint and index name in the extension settings.
## Setting up the index
### First install
Run the build command once. It creates a timestamped concrete index, immediately aliases your configured index name to it, and begins queuing documents. Search is available and improves as the queue processes:
```bash
composer require blomstra/search:*
php flarum blomstra:search:index build
php flarum queue:work
```
Enable the extension inside the admin area and configure the settings.
### Subsequent rebuilds (zero-downtime)
### Set up
When you need to rebuild the full index (e.g. after a mapping change):
Enable the extension in your admin area. Now to seed your existing discussions use the following command:
```bash
# Simple rebuild — promotes automatically once all jobs are queued
php flarum blomstra:search:index build
```
php flarum blomstra:search:index
# Or keep a backup of the old index in case you need to roll back
php flarum blomstra:search:index build --keep-backup
```
All mutations to discussions are automatically added and removed from the elasticsearch index.
After the queue drains, fill any gaps from content posted during the build:
### FAQ
```bash
php flarum blomstra:search:index fill
```
*I have another question.*
Reach out to us via https://helpdesk.blomstra.net. We will get back to you as soon as we can. If you have a running subscription please mention when you started your plan and/or which plan you are on. Always add sufficient information when reporting errors. We prefer errors being reported here, but understand that sometimes you can't.
If you kept a backup and want to roll back:
```bash
php flarum blomstra:search:index rollback
```
Once satisfied with the new index, drop the backup:
```bash
php flarum blomstra:search:index discard --backup
```
### Blue-green rebuild (manual promotion)
Use `--staging` to keep the old index live until you explicitly promote:
```bash
# 1. Build into a staging index — live index is untouched
php flarum blomstra:search:index build --staging
# 2. Drain the queue
php flarum queue:work --stop-when-empty
# 3. Promote the staging index to live
php flarum blomstra:search:index promote
# Or keep the old index as a backup:
php flarum blomstra:search:index promote --keep-backup
```
### Resuming or cancelling an interrupted build
If a build is interrupted, re-run it with the appropriate flag:
```bash
# Resume each seeder from where it left off
php flarum blomstra:search:index build --resume
# Drop the staging index and start completely fresh
php flarum blomstra:search:index build --fresh
# Cancel the build without starting a new one
php flarum blomstra:search:index discard --pending
```
### Filling gaps in an existing index
If documents are missing from the live index (e.g. due to queue failures):
```bash
php flarum blomstra:search:index fill
```
### Updating the mapping only
To push a mapping change to the live index without rebuilding:
```bash
php flarum blomstra:search:index mapping
```
## Command reference
| Command | Description |
|---|---|
| `build` | Rebuild the index and promote automatically once all jobs are queued. On first install, aliases immediately so search is live during seeding. |
| `build --keep-backup` | Rebuild and promote, retaining the old index as a backup for rollback. |
| `build --staging` | Build into a staging index without promoting — use `promote` when ready (blue-green workflow). |
| `build --resume` | Resume an interrupted build from where each seeder left off. |
| `build --fresh` | Drop the staging index and start completely fresh. |
| `promote` | Atomically swap the alias to the staging index. Prompts for confirmation (blue-green workflow). |
| `promote --keep-backup` | Promote and retain the replaced live index as a backup for rollback. |
| `rollback` | Restore the backup index to live. Deletes the index that was live. |
| `discard --pending` | Drop the staging index without promoting (cancels an in-progress build). |
| `discard --backup` | Drop the backup index (cleanup after `--keep-backup`). |
| `mapping` | Push updated mapping to the live index without rebuilding or reseeding. |
| `fill` | Seed only documents missing from the live index. |
| `build --only=discussions` | Seed only the specified document type (`discussions` or `posts`). |
| `build --throttle=N` | Wait N seconds between batches (reduces queue pressure). |
| `build --max-id=N` | Limit seeding to documents with ID ≤ N. |
| `promote --i-am-sure` | Skip the promotion confirmation prompt (for scripts and CI). |
## FAQ
## Queue configuration
*"Can I dispatch indexing jobs to a specific queue?"*
*Can I dispatch the sync jobs to another queue?*
Yes:
```php
\Blomstra\Search\Observe\Job::$onQueue = 'sync';
\Blomstra\Search\Jobs\Job::$onQueue = 'search';
```
*"I have a different question"*
Reach out ot us via https://support.on-floxum.com/t/ext-search . If you have an active subscription, please mention what plan you are on.
---
- Blomstra provides managed Flarum hosting.
- https://blomstra.net
- https://blomstra.community/t/ext-search
- Floxum provides managed Flarum hosting.
- https://floxum.com
- https://support.on-floxum.com/t/ext-search
Icon made by [Freepik](https://www.freepik.com) from [Flaticon](https://www.flaticon.com/).

View File

@ -1,5 +1,5 @@
{
"name": "blomstra/search",
"name": "sbp/flarum-search",
"description": "Replaces Flarum search with one powered by an elastic search server.",
"keywords": [
"extension",
@ -11,26 +11,27 @@
"elasticsearch"
],
"support": {
"email": "helpdesk@blomstra.net",
"email": "mueh+git@sbp.de",
"forum": "https://blomstra.community/t/ext-search"
},
"type": "flarum-extension",
"license": "MIT",
"authors": [
{
"name": "sbp",
"email": "info@sbp.de"
},
{
"name": "Team Blomstra",
"email": "development@blomstra.net",
"role": "Developer"
"role": "Original Developer"
}
],
"require": {
"php": ">= 8.1",
"flarum/core": "^2.0",
"elasticsearch/elasticsearch": "7.17.2",
"spatie/elasticsearch-query-builder": "^1.4.0"
},
"require-dev": {
"flarum/core": "^2.x-dev"
"php": ">= 8.0",
"flarum/core": "^1.2.0",
"elasticsearch/elasticsearch": "7.*",
"spatie/elasticsearch-query-builder": "^1.3.0"
},
"extra": {
"flarum-extension": {
@ -46,12 +47,7 @@
},
"autoload": {
"psr-4": {
"Blomstra\\Search\\": "src/"
}
},
"config": {
"allow-plugins": {
"php-http/discovery": true
"Sbp\\Search\\": "src/"
}
}
}

View File

@ -10,40 +10,32 @@
*
*/
namespace Blomstra\Search;
namespace Sbp\Search;
use Flarum\Discussion\Discussion as FlarumDiscussion;
use Flarum\Extend as Flarum;
use Flarum\Post\Post as FlarumPost;
return [
(new Flarum\ServiceProvider())->register(Provider::class),
(new Flarum\Frontend('forum'))
->js(__DIR__.'/js/dist/forum.js'),
(new Flarum\Frontend('admin'))
->js(__DIR__.'/js/dist/admin.js'),
->js(__DIR__.'/js/dist/admin.js')
->css(__DIR__.'/resources/less/admin.less'),
new Flarum\Locales(__DIR__.'/resources/locale'),
(new Flarum\ServiceProvider())
->register(Provider::class),
(new Flarum\Routes('api'))
->put('/blomstra/search/index', 'blomstra.search.index', Api\Controllers\IndexController::class),
->get('/sbp/search/{type}', 'sbp.search', Api\Controllers\SearchController::class)
->put('/sbp/search/index', 'sbp.search.index', Api\Controllers\IndexController::class),
(new Flarum\Console())
->command(Commands\BuildCommand::class),
(new Flarum\Settings())
->default('blomstra-search.search-discussion-subjects', true)
->default('blomstra-search.search-post-bodies', true),
(new Flarum\SearchDriver(Search\ElasticSearchDriver::class))
->addSearcher(FlarumDiscussion::class, Discussion\DiscussionSearcher::class)
->addFilter(Discussion\DiscussionSearcher::class, Discussion\PrivateFilterMutator::class)
->addMutator(Discussion\DiscussionSearcher::class, Discussion\PrivateFilterMutator::mutate(...))
->setFulltext(Discussion\DiscussionSearcher::class, Discussion\FulltextFilter::class),
(new Flarum\SearchIndex())
->indexer(FlarumDiscussion::class, Discussion\DiscussionIndexer::class)
->indexer(FlarumPost::class, Post\CommentPostIndexer::class),
->default('blomstra-search.search-post-bodies', true)
->default('blomstra-search.min-search-length', Commands\BuildCommand::DEFAULT_MIN_SEARCH_LENGTH)
->default('blomstra-search.stem-exclusion', '')
->serializeToForum('blomstraSearchMinLength', 'blomstra-search.min-search-length', 'intval')
->serializeToForum('blomstraSearchPostBodies', 'blomstra-search.search-post-bodies', fn ($v) => (bool) $v, true),
];

2
js/dist/admin.js vendored
View File

@ -1,2 +1,2 @@
(()=>{var a={n:e=>{var t=e&&e.__esModule?()=>e.default:()=>e;return a.d(t,{a:t}),t},d:(e,t)=>{for(var r in t)a.o(t,r)&&!a.o(e,r)&&Object.defineProperty(e,r,{enumerable:!0,get:t[r]})},o:(a,e)=>Object.prototype.hasOwnProperty.call(a,e),r:a=>{"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(a,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(a,"__esModule",{value:!0})}},e={};(()=>{"use strict";a.r(e);const t=flarum.core.compat["admin/app"];var r=a.n(t);r().initializers.add("blomstra-search",(function(){var a=new Map;["arabic","armenian","basque","bengali","brazilian","bulgarian","catalan","cjk","czech","danish","dutch","english","estonian","finnish","french","galician","german","greek","hindi","hungarian","indonesian","irish","italian","latvian","lithuanian","norwegian","persian","portuguese","romanian","russian","sorani","spanish","swedish","turkish","thai"].forEach((function(e){a.set(e,e)})),r().extensionData.for("blomstra-search").registerSetting({setting:"blomstra-search.elastic-endpoint",label:r().translator.trans("blomstra-search.admin.elastic-endpoint"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-username",label:r().translator.trans("blomstra-search.admin.elastic-username"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-password",label:r().translator.trans("blomstra-search.admin.elastic-password"),type:"password"}).registerSetting({setting:"blomstra-search.elastic-index",label:r().translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.analyzer-language",label:r().translator.trans("blomstra-search.admin.analyzer.label"),help:r().translator.trans("blomstra-search.admin.analyzer.help"),type:"select",options:Object.fromEntries(a.entries()),default:"english"}).registerSetting({setting:"blomstra-search.elastic-index",label:r().translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.search-discussion-subjects",label:r().translator.trans("blomstra-search.admin.search-discussion-subjects"),type:"switch"}).registerSetting({setting:"blomstra-search.search-post-bodies",label:r().translator.trans("blomstra-search.admin.search-post-bodies"),type:"switch"})}))})(),module.exports=e})();
(()=>{var a={n:t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return a.d(e,{a:e}),e},d:(t,e)=>{for(var n in e)a.o(e,n)&&!a.o(t,n)&&Object.defineProperty(t,n,{enumerable:!0,get:e[n]})},o:(a,t)=>Object.prototype.hasOwnProperty.call(a,t)};(()=>{"use strict";function t(a,e){return t=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(a,t){return a.__proto__=t,a},t(a,e)}const e=flarum.core.compat["admin/app"];var n=a.n(e);const r=flarum.core.compat["common/extend"],s=flarum.core.compat["admin/components/DashboardPage"];var i=a.n(s);const l=flarum.core.compat["admin/components/DashboardWidget"];var o=a.n(l);const c=flarum.core.compat["common/components/Alert"];var h=a.n(c),d=function(a){var e,r;function s(){return a.apply(this,arguments)||this}r=a,(e=s).prototype=Object.create(r.prototype),e.prototype.constructor=e,t(e,r);var i=s.prototype;return i.className=function(){return"ReindexWarningWidget"},i.content=function(){return m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle",title:n().translator.trans("blomstra-search.admin.reindex-required.title")},n().translator.trans("blomstra-search.admin.reindex-required.detail"))},s}(o());n().initializers.add("sbp-search",function(){var a=n().data.settings["blomstra-search.active-index"],t=n().data.settings["blomstra-search.index-compatible"];a&&"v3"!==t&&(0,r.extend)(i().prototype,"availableWidgets",function(a){a.add("blomstra-search-reindex",m(d),110)}),n().extensionData.for("sbp-search").registerSetting(function(){var t=n().data.settings["blomstra-search.indexed-analyzer"];if(!a||!t)return null;var e=this.setting("blomstra-search.analyzer-language")()||"english",r=this.setting("blomstra-search.stem-exclusion")()||"",s=n().data.settings["blomstra-search.indexed-stem-exclusion"]||"";return e===t&&r===s?null:m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle"},n().translator.trans("blomstra-search.admin.index-settings-changed"))}).registerSetting({setting:"blomstra-search.elastic-endpoint",label:n().translator.trans("blomstra-search.admin.elastic-endpoint"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-username",label:n().translator.trans("blomstra-search.admin.elastic-username"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-password",label:n().translator.trans("blomstra-search.admin.elastic-password"),type:"password"}).registerSetting({setting:"blomstra-search.elastic-index",label:n().translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.search-discussion-subjects",label:n().translator.trans("blomstra-search.admin.search-discussion-subjects"),type:"switch"}).registerSetting({setting:"blomstra-search.search-post-bodies",label:n().translator.trans("blomstra-search.admin.search-post-bodies"),type:"switch"}).registerSetting({setting:"blomstra-search.analyzer-language",label:n().translator.trans("blomstra-search.admin.analyzer.label"),help:n().translator.trans("blomstra-search.admin.analyzer.help"),type:"select",options:{arabic:"Arabic",armenian:"Armenian",basque:"Basque",bengali:"Bengali",brazilian:"Brazilian Portuguese",bulgarian:"Bulgarian",catalan:"Catalan",cjk:"CJK (Chinese, Japanese, Korean)",czech:"Czech",danish:"Danish",dutch:"Dutch",english:"English",estonian:"Estonian",finnish:"Finnish",french:"French",galician:"Galician",german:"German",greek:"Greek",hindi:"Hindi",hungarian:"Hungarian",indonesian:"Indonesian",irish:"Irish",italian:"Italian",latvian:"Latvian",lithuanian:"Lithuanian",norwegian:"Norwegian",persian:"Persian",portuguese:"Portuguese",romanian:"Romanian",russian:"Russian",sorani:"Sorani (Kurdish)",spanish:"Spanish",standard:"Standard (generic)",swedish:"Swedish",turkish:"Turkish",thai:"Thai"},default:"english"}).registerSetting(function(){var a="cjk"===(this.setting("blomstra-search.analyzer-language")()||"english");return this.buildSettingComponent({setting:"blomstra-search.stem-exclusion",type:"textarea",label:n().translator.trans("blomstra-search.admin.settings.stem-exclusion.label"),help:n().translator.trans("blomstra-search.admin.settings.stem-exclusion.help"),disabled:a})}).registerSetting(function(){var a="cjk"===(this.setting("blomstra-search.analyzer-language")()||"english");return this.buildSettingComponent({setting:"blomstra-search.min-search-length",label:n().translator.trans("blomstra-search.admin.min-search-length.label"),help:n().translator.trans("blomstra-search.admin.min-search-length.help"),type:"select",options:{1:"1",2:"2",3:"3",4:"4"},default:n().data.settings["blomstra-search.min-search-length"],disabled:a})})})})(),module.exports={}})();
//# sourceMappingURL=admin.js.map

File diff suppressed because one or more lines are too long

3
js/dist/forum.js vendored

File diff suppressed because one or more lines are too long

1
js/dist/forum.js.LICENSE.txt vendored Normal file
View File

@ -0,0 +1 @@
/*! regenerator-runtime -- Copyright (c) 2014-present, Facebook, Inc. -- license (MIT): https://github.com/babel/babel/blob/main/packages/babel-helpers/LICENSE */

File diff suppressed because one or more lines are too long

1885
js/package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@ -1,49 +1,98 @@
import app from 'flarum/admin/app';
import { extend } from 'flarum/common/extend';
import DashboardPage from 'flarum/admin/components/DashboardPage';
import DashboardWidget from 'flarum/admin/components/DashboardWidget';
import Alert from 'flarum/common/components/Alert';
app.initializers.add('blomstra-search', () => {
const languages = new Map();
[
'arabic',
'armenian',
'basque',
'bengali',
'brazilian',
'bulgarian',
'catalan',
'cjk',
'czech',
'danish',
'dutch',
'english',
'estonian',
'finnish',
'french',
'galician',
'german',
'greek',
'hindi',
'hungarian',
'indonesian',
'irish',
'italian',
'latvian',
'lithuanian',
'norwegian',
'persian',
'portuguese',
'romanian',
'russian',
'sorani',
'spanish',
'swedish',
'turkish',
'thai',
].forEach((language) => {
languages.set(language, language);
const REQUIRED_INDEX_COMPAT = 'v3';
class ReindexWarningWidget extends DashboardWidget {
className() {
return 'ReindexWarningWidget';
}
content() {
return m(
Alert,
{
type: 'warning',
dismissible: false,
icon: 'fas fa-exclamation-triangle',
title: app.translator.trans('blomstra-search.admin.reindex-required.title'),
},
app.translator.trans('blomstra-search.admin.reindex-required.detail')
);
}
}
app.initializers.add('sbp-search', () => {
const activeIndex = app.data.settings['blomstra-search.active-index'];
const compatVersion = app.data.settings['blomstra-search.index-compatible'];
if (activeIndex && compatVersion !== REQUIRED_INDEX_COMPAT) {
extend(DashboardPage.prototype, 'availableWidgets', function (items) {
items.add('blomstra-search-reindex', m(ReindexWarningWidget), 110);
});
}
const languages: Record<string, string> = {
arabic: 'Arabic',
armenian: 'Armenian',
basque: 'Basque',
bengali: 'Bengali',
brazilian: 'Brazilian Portuguese',
bulgarian: 'Bulgarian',
catalan: 'Catalan',
cjk: 'CJK (Chinese, Japanese, Korean)',
czech: 'Czech',
danish: 'Danish',
dutch: 'Dutch',
english: 'English',
estonian: 'Estonian',
finnish: 'Finnish',
french: 'French',
galician: 'Galician',
german: 'German',
greek: 'Greek',
hindi: 'Hindi',
hungarian: 'Hungarian',
indonesian: 'Indonesian',
irish: 'Irish',
italian: 'Italian',
latvian: 'Latvian',
lithuanian: 'Lithuanian',
norwegian: 'Norwegian',
persian: 'Persian',
portuguese: 'Portuguese',
romanian: 'Romanian',
russian: 'Russian',
sorani: 'Sorani (Kurdish)',
spanish: 'Spanish',
standard: 'Standard (generic)',
swedish: 'Swedish',
turkish: 'Turkish',
thai: 'Thai',
};
app.extensionData
.for('blomstra-search')
.for('sbp-search')
.registerSetting(function (this: any) {
// No index yet, or first build predates this tracking — stay silent.
const indexedAnalyzer = app.data.settings['blomstra-search.indexed-analyzer'];
if (!activeIndex || !indexedAnalyzer) return null;
const currentAnalyzer = this.setting('blomstra-search.analyzer-language')() || 'english';
const currentStemExclusion = this.setting('blomstra-search.stem-exclusion')() || '';
const indexedStemExclusion = app.data.settings['blomstra-search.indexed-stem-exclusion'] || '';
if (currentAnalyzer === indexedAnalyzer && currentStemExclusion === indexedStemExclusion) return null;
return m(
Alert,
{ type: 'warning', dismissible: false, icon: 'fas fa-exclamation-triangle' },
app.translator.trans('blomstra-search.admin.index-settings-changed')
);
})
.registerSetting({
setting: 'blomstra-search.elastic-endpoint',
label: app.translator.trans('blomstra-search.admin.elastic-endpoint'),
@ -65,20 +114,6 @@ app.initializers.add('blomstra-search', () => {
default: 'flarum',
type: 'input',
})
.registerSetting({
setting: 'blomstra-search.analyzer-language',
label: app.translator.trans('blomstra-search.admin.analyzer.label'),
help: app.translator.trans('blomstra-search.admin.analyzer.help'),
type: 'select',
options: Object.fromEntries(languages.entries()),
default: 'english',
})
.registerSetting({
setting: 'blomstra-search.elastic-index',
label: app.translator.trans('blomstra-search.admin.elastic-index'),
default: 'flarum',
type: 'input',
})
.registerSetting({
setting: 'blomstra-search.search-discussion-subjects',
label: app.translator.trans('blomstra-search.admin.search-discussion-subjects'),
@ -88,5 +123,35 @@ app.initializers.add('blomstra-search', () => {
setting: 'blomstra-search.search-post-bodies',
label: app.translator.trans('blomstra-search.admin.search-post-bodies'),
type: 'switch',
})
.registerSetting({
setting: 'blomstra-search.analyzer-language',
label: app.translator.trans('blomstra-search.admin.analyzer.label'),
help: app.translator.trans('blomstra-search.admin.analyzer.help'),
type: 'select',
options: languages,
default: 'english',
})
.registerSetting(function (this: any) {
const isCjk = (this.setting('blomstra-search.analyzer-language')() || 'english') === 'cjk';
return this.buildSettingComponent({
setting: 'blomstra-search.stem-exclusion',
type: 'textarea',
label: app.translator.trans('blomstra-search.admin.settings.stem-exclusion.label'),
help: app.translator.trans('blomstra-search.admin.settings.stem-exclusion.help'),
disabled: isCjk,
});
})
.registerSetting(function (this: any) {
const isCjk = (this.setting('blomstra-search.analyzer-language')() || 'english') === 'cjk';
return this.buildSettingComponent({
setting: 'blomstra-search.min-search-length',
label: app.translator.trans('blomstra-search.admin.min-search-length.label'),
help: app.translator.trans('blomstra-search.admin.min-search-length.help'),
type: 'select',
options: { '1': '1', '2': '2', '3': '3', '4': '4' },
default: app.data.settings['blomstra-search.min-search-length'],
disabled: isCjk,
});
});
});

View File

@ -0,0 +1,33 @@
import app from 'flarum/forum/app';
import { override } from 'flarum/common/extend';
import DiscussionListState from 'flarum/forum/states/DiscussionListState';
export default function extendDiscussionState() {
override(DiscussionListState.prototype, 'loadPage', async function (this: DiscussionListState, original, page: number = 1) {
// If existing payload is given or no search is made, fallback on native page.
if (!this.requestParams()?.filter?.q) return original.call(this, page);
const params = this.requestParams();
params.page = {
offset: this.pageSize * (page - 1),
...params.page,
};
if (Array.isArray(params.include)) {
params.include = params.include.join(',');
}
// Construct API search URI
const url = `${app.forum.attribute('apiUrl')}/sbp/search/${this.type}`;
// Make API GET request
const results = await app.request({ params, url, method: 'GET' });
// Parse API response into models and push to store
return app.store.pushPayload(results);
});
}

View File

@ -0,0 +1,79 @@
import app from 'flarum/forum/app';
import highlight from 'flarum/common/helpers/highlight';
import LinkButton from 'flarum/common/components/LinkButton';
import Link from 'flarum/common/components/Link';
import { SearchSource } from 'flarum/forum/components/Search';
import type Mithril from 'mithril';
/**
* The `DiscussionsSearchSource` finds and displays discussion search results in
* the search dropdown.
*/
export default class DiscussionsSearchSource implements SearchSource {
/**
* Map of lowercase search queries to their respective model results
*/
protected results = new Map<string, any[]>();
/**
* Model name used for assembling API endpoint URL, and for frontend DOM.
*/
private type = 'discussions';
async search(query: string): Promise<void> {
// Suppress ES instant results when "search within comments" is ON;
// only usernames are shown in that mode.
query = query.toLowerCase();
this.results.set(query, []);
const params = {
filter: { q: query, autocomplete: 1 },
page: { limit: 3 },
include: 'mostRelevantPost',
};
// Construct API search URI
const url = `${app.forum.attribute('apiUrl')}/sbp/search/${this.type}`;
// Make API GET request
const results = await app.request({ params, url, method: 'GET' });
// Parse API response into models and push to store
const models = app.store.pushPayload(results);
// Add models to results map
this.results.set(query, models);
}
view(query: string): Array<Mithril.Vnode> {
query = query.toLowerCase();
// Get results from map
const queryResults = this.results.get(query) || [];
const results = queryResults.map((discussion: any) => {
const mostRelevantPost = discussion.mostRelevantPost();
return (
<li className="DiscussionSearchResult" data-index={`${this.type}${discussion.id()}`}>
<Link href={app.route.discussion(discussion, mostRelevantPost && mostRelevantPost.number())}>
<div className="DiscussionSearchResult-title">{highlight(discussion.title(), query)}</div>
{!!mostRelevantPost && <div className="DiscussionSearchResult-excerpt">{highlight(mostRelevantPost.contentPlain(), query, 100)}</div>}
</Link>
</li>
);
});
return [
<li className="Dropdown-header">{app.translator.trans('core.forum.search.discussions_heading')}</li>,
<li>
<LinkButton icon="fas fa-search" href={app.route('index', { q: query })}>
{app.translator.trans('core.forum.search.all_discussions_button', { query })}
</LinkButton>
</li>,
...results,
];
}
}

View File

@ -0,0 +1,26 @@
import { extend } from 'flarum/common/extend';
import DiscussionListItem from 'flarum/forum/components/DiscussionListItem';
import TerminalPost from 'flarum/forum/components/TerminalPost';
export default function extendDiscussionListItem() {
extend(DiscussionListItem.prototype, 'infoItems', function (items) {
const params = this.attrs.params;
if (!params.q) return;
const hasFieldSort = params.sort && params.sort !== 'relevance';
if (hasFieldSort) {
// Field sort active (latest, oldest, top, …): replace excerpt with TerminalPost.
// An excerpt is meaningless when results are ordered by date/count rather than relevance.
items.remove('excerpt');
if (!items.has('terminalPost')) {
items.add('terminalPost', <TerminalPost discussion={this.attrs.discussion} lastPost={!this.showFirstPost()} />);
}
} else if (!items.has('excerpt')) {
// Relevance mode but no excerpt (mostRelevantPost was null or non-comment type).
// Fall back to TerminalPost so the info section is never silently empty.
items.add('terminalPost', <TerminalPost discussion={this.attrs.discussion} lastPost={true} />);
}
});
}

View File

@ -1,5 +1,32 @@
import app from 'flarum/forum/app';
import Search, { SearchAttrs, SearchSource } from 'flarum/forum/components/Search';
import { extend } from 'flarum/common/extend';
import ItemList from 'flarum/common/utils/ItemList';
import DiscussionsSearchSource from './SearchSources/DiscussionsSearchSource';
import extendDiscussionState from './PaginatedListStates/extendDiscussionState';
import extendDiscussionListItem from './extendDiscussionListItem';
app.initializers.add('blomstra-search', () => {
//
extend(Search.prototype, 'sourceItems', function (this: Search<SearchAttrs>, items: ItemList<SearchSource>) {
// app.forum is not available during initializers (it is set after they run),
// so read the setting lazily here, at first render time.
const minLength = (app.forum.attribute('blomstraSearchMinLength') as number) || Search.MIN_SEARCH_LEN;
if (minLength !== Search.MIN_SEARCH_LEN) {
(Search as any).MIN_SEARCH_LEN = minLength;
}
items.replace('discussions', new DiscussionsSearchSource());
});
});
app.initializers.add(
'blomstra-search-early',
() => {
extendDiscussionState();
extendDiscussionListItem();
},
999999
);

View File

@ -0,0 +1,14 @@
diff --git a/src/Commands/BuildCommand.php b/src/Commands/BuildCommand.php
index 9024e1e..9024e1e 100644
--- a/src/Commands/BuildCommand.php
+++ b/src/Commands/BuildCommand.php
@@ -38,6 +38,6 @@ class BuildCommand extends Command
/** Matches Flarum's Search::MIN_SEARCH_LEN — the default minimum query length. */
public const DEFAULT_MIN_SEARCH_LENGTH = 3;
/** N-gram bounds for the title autocomplete subfield (not admin-configurable). */
- private const TITLE_MIN_GRAM = 2;
- private const TITLE_MAX_GRAM = 15;
+ private const TITLE_MIN_GRAM = 3;
+ private const TITLE_MAX_GRAM = 20;
protected $signature = 'blomstra:search:index

View File

@ -0,0 +1,3 @@
.ReindexWarningWidget {
padding: 0;
}

View File

@ -8,8 +8,21 @@ blomstra-search:
label: Analyzer language
help: |
The analyzer makes search understand stop words and undertakes language
specific improvements for indexing.
specific improvements for indexing. Changing this requires a full index rebuild to take effect.
index-settings-changed: "Search index settings have changed. Run: php flarum blomstra:search:index build"
reindex-required:
title: Search index rebuild required
detail: "The search index is not compatible with this version of the extension. Run: php flarum blomstra:search:index build"
search-discussion-subjects: Search inside discussion titles
search-post-bodies: Search inside comments
match-sentences: Match search term against full sentence
match-words: Match search term against full words
settings:
stem-exclusion:
label: "Protected words (stem exclusion)"
help: "One word per line. These words will not be stemmed during search (e.g. brand names, proper nouns). Not applicable for CJK or Thai languages. Requires a full re-index to take effect."
min-search-length:
label: Minimum search query length
help: |
Minimum number of characters required before a search is triggered. Lower this to 1 or 2
for CJK (Chinese, Japanese, Korean) communities where single characters carry full meaning.

28
src/Api/Client.php Normal file
View File

@ -0,0 +1,28 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Api;
use Illuminate\Support\Arr;
use Psr\Http\Message\ResponseInterface;
class Client extends \Flarum\Api\Client
{
public function get(string $path): ResponseInterface
{
if ($path === '/discussions' && Arr::has($this->queryParams, 'filter.q')) {
return parent::get('/blomstra/search/discussions');
}
return parent::get($path);
}
}

View File

@ -10,9 +10,9 @@
*
*/
namespace Blomstra\Search\Api\Controllers;
namespace Sbp\Search\Api\Controllers;
use Blomstra\Search\Commands\BuildCommand;
use Sbp\Search\Commands\BuildCommand;
use Laminas\Diactoros\Response\EmptyResponse;
use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\ServerRequestInterface;

View File

@ -10,14 +10,15 @@
*
*/
namespace Blomstra\Search\Api\Controllers;
namespace Sbp\Search\Api\Controllers;
use Blomstra\Search\Elasticsearch\Builder;
use Blomstra\Search\Elasticsearch\MatchPhraseQuery;
use Blomstra\Search\Elasticsearch\MatchQuery;
use Blomstra\Search\Elasticsearch\TermsQuery;
use Blomstra\Search\Save\Document as ElasticDocument;
use Blomstra\Search\Searchers\Searcher;
use Sbp\Search\Elasticsearch\HasChildQuery;
use Sbp\Search\Elasticsearch\MatchPhraseQuery;
use Sbp\Search\Elasticsearch\MatchQuery;
use Sbp\Search\Elasticsearch\TermsQuery;
use Sbp\Search\Searchers\CommentPostSearcher;
use Sbp\Search\Searchers\DiscussionSearcher;
use Sbp\Search\Searchers\Searcher;
use Elasticsearch\Client;
use Flarum\Api\Controller\ListDiscussionsController;
use Flarum\Api\Serializer\DiscussionSerializer;
@ -27,14 +28,16 @@ use Flarum\Group\Group;
use Flarum\Http\RequestUtil;
use Flarum\Http\UrlGenerator;
use Flarum\Settings\SettingsRepositoryInterface;
use Flarum\Tags\Tag;
use Flarum\User\User;
use Illuminate\Contracts\Container\Container;
use Illuminate\Support\Arr;
use Illuminate\Support\Collection;
use Illuminate\Support\Str;
use Psr\Http\Message\ServerRequestInterface;
use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
use Psr\Log\LoggerInterface;
use Spatie\ElasticsearchQueryBuilder\Builder;
use Sbp\Search\Elasticsearch\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery;
use Spatie\ElasticsearchQueryBuilder\Sorts\Sort;
use Tobscure\JsonApi\Document;
@ -47,109 +50,133 @@ class SearchController extends ListDiscussionsController
'lastPostedAt' => 'updated_at',
'createdAt' => 'created_at',
'commentCount' => 'comment_count',
'view_count' => 'view_count',
];
protected Collection $searchers;
protected bool $matchSentences;
protected bool $matchWords;
protected ?Searcher $discussionSearcher;
protected ?Searcher $postSearcher;
public function __construct(protected Client $elastic, protected UrlGenerator $uri, Container $container, SettingsRepositoryInterface $settings)
{
$this->searchers = $this->gatherSearchers($container->tagged('blomstra.search.searchers'));
$this->matchSentences = true;
$this->matchWords = true;
}
protected function gatherSearchers(iterable $searchers)
{
return collect($searchers)
->map(fn ($searcher) => new $searcher())
->filter(fn (Searcher $searcher) => $searcher->enabled());
$searchers = collect($container->tagged('blomstra.search.searchers'));
$this->discussionSearcher = $searchers->first(fn ($s) => $s instanceof DiscussionSearcher);
$this->postSearcher = $searchers->first(fn ($s) => $s instanceof CommentPostSearcher);
}
protected function data(ServerRequestInterface $request, Document $document)
{
// Not used for now.
$type = Arr::get($request->getQueryParams(), 'type');
$actor = RequestUtil::getActor($request);
$filters = $this->extractFilter($request);
$search = $this->getSearch($filters);
$limit = $this->extractLimit($request);
$offset = $this->extractOffset($request);
$include = array_merge($this->extractInclude($request), ['state']);
$filterQuery = BoolQuery::create();
// Autocomplete path: fast prefix match on title.autocomplete only.
if (!empty($filters['autocomplete']) && !empty($search)) {
return $this->handleAutocomplete($search, $actor, $limit, $offset, $include, $document, $request);
}
$knownSortFields = array_merge(array_values($this->translateSort), ['rawId']);
$logger = resolve(LoggerInterface::class);
$phpSortField = null;
$phpSortDir = 'desc';
$needsScoring = true;
$sorts = [];
foreach ($this->extractSort($request) as $field => $direction) {
$translated = $this->translateSort[$field] ?? $field;
if (!in_array($translated, $knownSortFields)) {
$logger->warning("blomstra/search: unknown sort field \"{$field}\", ignoring.");
continue;
}
$sorts[] = new Sort($translated, $direction);
$needsScoring = false;
if ($phpSortField === null && $translated !== 'rawId') {
$phpSortField = $translated;
$phpSortDir = $direction;
}
}
// Default to latest when no explicit sort is requested. This lets has_child use
// score_mode:none, which skips child scoring entirely and lets ES short-circuit
// early on large corpora. Relevance is still available via sort=relevant if added.
if (empty($sorts)) {
$needsScoring = false;
$phpSortField = 'updated_at';
$phpSortDir = 'desc';
$sorts[] = new Sort('updated_at', 'desc');
}
$query = BoolQuery::create()
// Always restrict to discussion documents; posts are only searched via has_child.
->add(TermQuery::create('join_field', 'discussion'), 'filter');
if (!empty($search)) {
if ($this->matchSentences) {
$filterQuery->add($this->sentenceMatch($search));
}
if ($this->matchWords) {
$filterQuery->add($this->wordMatch($search, 'and'));
}
if ($this->matchWords) {
$filterQuery->add($this->wordMatch($search, 'or'));
}
$query->add($this->buildTextQuery($search, $actor, $needsScoring));
}
$this->addFilters($query, $actor, $filters);
$builder = (new Builder($this->elastic))
->index(resolve('blomstra.search.elastic_index'))
->size($limit + 1)
->from($this->extractOffset($request))
->addQuery(
$this->addFilters($filterQuery, $actor, $filters)
);
->addQuery($query);
foreach ($this->extractSort($request) as $field => $direction) {
$field = $this->translateSort[$field] ?? $field;
$builder->addSort(new Sort($field, $direction));
foreach ($sorts as $sort) {
$builder->addSort($sort);
}
$response = $builder->search();
// track_total_hits: false lets ES stop counting once it has collected
// enough results in sort order, avoiding a full-index count on every query.
$payload = $builder->getPayload();
$payload['track_total_hits'] = false;
$response = $this->elastic->search([
'index' => resolve('blomstra.search.elastic_index'),
'size' => $limit + 1,
'from' => $offset,
'body' => $payload,
]);
Discussion::setStateUser($actor);
// Eager load groups for use in the policies (isAdmin check)
if (in_array('mostRelevantPost.user', $include)) {
$include[] = 'mostRelevantPost.user.groups';
// If the first level of the relationship wasn't explicitly included,
// add it so the code below can look for it
if (!in_array('mostRelevantPost', $include)) {
$include[] = 'mostRelevantPost';
}
}
// we need to retrieve all discussion ids and when the results are posts,
// their ids as most relevant post id
// All hits are discussion documents. Extract the best-matching post ID from
// inner_hits when a has_child clause matched (i.e. the match came from a post).
$results = Collection::make(Arr::get($response, 'hits.hits'))
->map(function ($hit) {
$type = $hit['_source']['type'];
$id = Str::after($hit['_source']['id'], "$type:");
// _id is "discussions:123" — parse the numeric part directly.
$discussionId = Str::after($hit['_id'], 'discussions:');
// rawId on the inner hit gives us the integer post ID.
$bestPostId = Arr::get($hit, 'inner_hits.best_post.hits.hits.0._source.rawId');
if ($type === 'posts') {
return [
'most_relevant_post_id' => $id,
'weight' => Arr::get($hit, 'sort.0'),
'discussion_id' => $discussionId,
'most_relevant_post_id' => $bestPostId,
'weight' => Arr::get($hit, 'sort.0', Arr::get($hit, '_score', 0)),
];
} else {
return [
'discussion_id' => $id,
'weight' => Arr::get($hit, 'sort.0'),
];
}
});
$document->addPaginationLinks(
$this->uri->to('api')->route('blomstra.search', [
'type' => 'discussions',
]),
$this->uri->to('api')->route('sbp.search', ['type' => 'discussions']),
$request->getQueryParams(),
$offset,
$limit,
@ -159,28 +186,25 @@ class SearchController extends ListDiscussionsController
$results = $results->take($limit);
$discussions = Discussion::query()
->select('discussions.*')
->join('posts', 'posts.discussion_id', 'discussions.id')
// Extra safety to prevent leaking hidden discussion (titles) towards search results.
->when($actor->isGuest() || !$actor->hasPermission('discussion.hide'), fn ($query) => $query->whereNull('discussions.hidden_at'))
->where(function ($query) use ($results) {
$query
->whereIn('discussions.id', $results->pluck('discussion_id')->filter())
->orWhereIn('posts.id', $results->pluck('most_relevant_post_id')->filter());
})
->when(
$actor->isGuest() || !$actor->hasPermission('discussion.hide'),
fn ($q) => $q->whereNull('hidden_at')
)
->whereIn('id', $results->pluck('discussion_id')->filter())
->get()
->each(function (Discussion $discussion) use ($results) {
if (in_array($discussion->id, $results->pluck('discussion_id')->toArray())) {
$discussion->most_relevant_post_id = $discussion->first_post_id;
$discussion->weight = $results->firstWhere('discussion_id', $discussion->id)['weight'] ?? 0;
} else {
$post = $discussion->posts()->whereIn('id', $results->pluck('most_relevant_post_id'))->first();
$discussion->most_relevant_post_id = $post?->id ?? $discussion->first_post_id;
$discussion->weight = $results->firstWhere('most_relevant_post_id', $post?->id)['weight'] ?? 0;
}
$result = $results->firstWhere('discussion_id', $discussion->id);
$discussion->most_relevant_post_id = $result['most_relevant_post_id']
?? $discussion->first_post_id;
$discussion->weight = $result['weight'] ?? 0;
})
->keyBy('id')
->sortByDesc('weight')
->when(
$phpSortField,
fn ($c) => $phpSortDir === 'desc' ? $c->sortByDesc($phpSortField) : $c->sortBy($phpSortField),
fn ($c) => $c->sortByDesc('weight')
)
->unique();
$this->loadRelations($discussions, $include);
@ -198,33 +222,208 @@ class SearchController extends ListDiscussionsController
return $discussions;
}
protected function getDocument(string $type): ?ElasticDocument
{
$documents = resolve(Container::class)->tagged('blomstra.search.documents');
/**
* Fast autocomplete path: match prefix n-grams on title.autocomplete only.
* No has_child, no phrase scoring a single MatchQuery suffices.
*/
protected function handleAutocomplete(
string $search,
User $actor,
int $limit,
int $offset,
array $include,
Document $document,
ServerRequestInterface $request
): mixed {
$query = BoolQuery::create()
->add(TermQuery::create('join_field', 'discussion'), 'filter');
return collect($documents)->first(function (ElasticDocument $document) use ($type) {
return $document->type() === $type;
});
$query->add($this->buildAutocompleteQuery($search), 'must');
$this->addFilters($query, $actor, []);
$payload = (new Builder($this->elastic))
->index(resolve('blomstra.search.elastic_index'))
->addQuery($query)
->addSort(new Sort('updated_at', 'desc'))
->getPayload();
$response = $this->elastic->search([
'index' => resolve('blomstra.search.elastic_index'),
'size' => $limit + 1,
'from' => $offset,
'body' => $payload,
]);
Discussion::setStateUser($actor);
$results = Collection::make(Arr::get($response, 'hits.hits'))
->map(fn ($hit) => [
'discussion_id' => Str::after($hit['_id'], 'discussions:'),
'most_relevant_post_id' => null,
'weight' => Arr::get($hit, 'sort.0', 0),
]);
$document->addPaginationLinks(
$this->uri->to('api')->route('sbp.search', ['type' => 'discussions']),
$request->getQueryParams(),
$offset,
$limit,
$results->count() > $limit ? null : 0
);
$results = $results->take($limit);
$discussions = Discussion::query()
->when(
$actor->isGuest() || !$actor->hasPermission('discussion.hide'),
fn ($q) => $q->whereNull('hidden_at')
)
->whereIn('id', $results->pluck('discussion_id')->filter())
->get()
->each(function (Discussion $discussion) use ($results) {
$result = $results->firstWhere('discussion_id', $discussion->id);
$discussion->most_relevant_post_id = $result['most_relevant_post_id'] ?? $discussion->first_post_id;
$discussion->weight = $result['weight'] ?? 0;
})
->sortByDesc('updated_at')
->unique();
$this->loadRelations($discussions, $include);
return $discussions;
}
protected function buildAutocompleteQuery(string $search): BoolQuery
{
$query = BoolQuery::create();
// Title prefix match (stronger)
$query->add(
(new MatchQuery('title.autocomplete', $search))
->operator('and')
->boost(1.0),
'should'
);
// Content prefix match (weaker)
$query->add(
(new MatchQuery('content.autocomplete', $search))
->operator('and')
->boost(0.4),
'should'
);
$query->minimumShouldMatch(1);
return $query;
}
/**
* Build the text-matching portion of the query.
*
* Discussion titles are matched directly (filtered to join_field=discussion).
* Post bodies are matched via has_child. When $needsScoring is true (no explicit
* sort, i.e. relevance ordering), score_mode=sum accumulates child scores onto the
* parent so that discussions with many strongly-matching posts rank higher. When
* false (any explicit field sort), score_mode=none skips scoring entirely ES only
* checks whether a matching child exists, which is significantly cheaper on large
* corpora.
* inner_hits returns the best-matching post for use as mostRelevantPost.
* Hidden posts are only included in matching for users with post.hide permission.
*/
protected function buildTextQuery(string $search, User $actor, bool $needsScoring = false): BoolQuery
{
$textQuery = BoolQuery::create();
// Strong direct title match for the full search page.
$textQuery->add(
(new MatchQuery('title', $search))
->operator('and')
->boost(4.0),
'should'
);
// Low-weight prefix match on title.autocomplete for partial title hits.
$textQuery->add(
(new MatchQuery('title.autocomplete', $search))
->operator('and')
->boost(0.35),
'should'
);
// Low-weight prefix match on title.autocomplete for the full search page.
// Exact / normal title matches should still rank higher than prefix-only hits.
$textQuery->add(
(new MatchQuery('title.autocomplete', $search))
->operator('and')
->boost(0.35),
'should'
);
if ($this->discussionSearcher?->enabled()) {
$textQuery->add($this->buildShouldClauses($search, $this->discussionSearcher->boost()), 'should');
}
if ($this->postSearcher?->enabled()) {
$postQuery = $this->buildShouldClauses($search, $this->postSearcher->boost());
// Guests and non-moderators may not see hidden posts; exclude them from child matching.
if ($actor->isGuest() || !$actor->hasPermission('post.hide')) {
$postQuery->add(TermQuery::create('is_hidden', 'false'), 'filter');
}
// Without minimum_should_match, ES default MSM is 0 when a filter clause is present,
// causing has_child to score every non-hidden post instead of only matching ones.
$postQuery->minimumShouldMatch(1);
$textQuery->add(
HasChildQuery::create('post', $postQuery, $needsScoring ? 'sum' : 'none')->withInnerHits(),
'should'
);
}
return $textQuery;
}
protected function buildShouldClauses(string $search, float $boost): BoolQuery
{
$should = BoolQuery::create();
if ($this->matchSentences) {
$should->add((new MatchPhraseQuery('content', $search))->boost(2 * $boost), 'should');
}
if ($this->matchWords) {
$should->add((new MatchQuery('content', $search))->operator('and')->boost(1.8 * $boost), 'should');
$should->add((new MatchQuery('content.autocomplete', $search))->operator('and')->boost(0.30 * $boost), 'should');
}
return $should;
}
protected function extensionEnabled(string $extension): bool
{
/** @var ExtensionManager $manager */
$manager = resolve(ExtensionManager::class);
return $manager->isEnabled($extension);
return resolve(ExtensionManager::class)->isEnabled($extension);
}
protected function addFilters(BoolQuery $query, User $actor, array $filters = []): BoolQuery
protected function addFilters(BoolQuery $query, User $actor, array $filters = []): void
{
$groups = $this->getGroups($actor);
$onlyPrivate = Str::contains($filters['q'] ?? '', 'is:private');
$subQuery = BoolQuery::create()
->add(TermQuery::create('is_private', 'false'))
->add(TermsQuery::create('groups', $groups->toArray()));
if ($this->extensionEnabled('flarum-tags') && !empty($filters['tag'])) {
$slugs = is_array($filters['tag']) ? $filters['tag'] : explode(',', $filters['tag']);
$tagIds = Tag::query()->whereIn('slug', $slugs)->pluck('id')->toArray();
if (!empty($tagIds)) {
$query->add(TermsQuery::create('tags', $tagIds), 'filter');
}
}
if ($this->extensionEnabled('fof-byobu') && $actor->exists) {
$byobuQuery = BoolQuery::create()
->add(TermQuery::create('is_private', 'true'))
@ -243,55 +442,12 @@ class SearchController extends ListDiscussionsController
}
}
$query->add(
$subQuery,
'filter'
);
return $query;
}
protected function boolQuery(Query $parent, float $boost = 1): Query
{
$bool = new BoolQuery();
/** @var Searcher $searcher */
foreach ($this->searchers as $searcher) {
$searcher = new $searcher();
$bool->add(
BoolQuery::create()
->add(TermQuery::create('type', $searcher->type()), 'filter')
->add(clone $parent->boost($boost * $searcher->boost())),
'should'
);
}
return $bool;
}
protected function sentenceMatch(string $q): Query
{
$query = (new MatchPhraseQuery('content', $q));
return $this->boolQuery($query, 2);
}
protected function wordMatch(string $q, string $operator = 'or'): Query
{
$query = (new MatchQuery('content', $q))
->operator($operator);
$boost = $operator === 'and' ? 1.8 : .8;
return $this->boolQuery($query, $boost);
$query->add($subQuery, 'filter');
}
protected function getGroups(User $actor): Collection
{
/** @var Collection $groups */
$groups = $actor->groups->pluck('id');
$groups->add(Group::GUEST_ID);
if ($actor->is_email_confirmed) {
@ -306,10 +462,12 @@ class SearchController extends ListDiscussionsController
$search = Arr::get($filters, 'q');
if ($search) {
// Strip Flarum gambit operators (tag:foo, author:bar, is:private, etc.)
// before passing to ES. These are structural filters handled separately;
// leaving them in causes operator:and to require the gambit tokens to
// appear literally in post content, producing zero results.
$q = collect(explode(' ', $search))
->filter(function (string $part) {
return $part !== 'is:private';
})
->filter(fn (string $part) => !preg_match('/^\w+:/', $part))
->filter()
->join(' ');

View File

@ -10,122 +10,389 @@
*
*/
namespace Blomstra\Search\Commands;
namespace Sbp\Search\Commands;
use Blomstra\Search\Discussion\DiscussionIndexer;
use Blomstra\Search\Elasticsearch\Builder;
use Blomstra\Search\Post\CommentPostIndexer;
use Sbp\Search\Jobs\Job;
use Sbp\Search\Jobs\UpdateSearchJob;
use Sbp\Search\Seeders\Seeder;
use Elasticsearch\Client;
use Flarum\Discussion\Discussion;
use Flarum\Post\Post;
use Flarum\Search\Job\IndexJob;
use Elasticsearch\Common\Exceptions\ElasticsearchException;
use Flarum\Settings\SettingsRepositoryInterface;
use Illuminate\Console\Command;
use Illuminate\Contracts\Container\Container;
use Illuminate\Contracts\Queue\Queue;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Support\Arr;
use Spatie\ElasticsearchQueryBuilder\Builder;
use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\RangeQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery;
class BuildCommand extends Command
{
/**
* Bump this when a mapping change requires a full reindex.
* Written into the mapping's _meta.index_compat_version on every build and
* read back into blomstra-search.index-compatible by saveIndexedConfig so the
* value always reflects the live index including after a promote/rollback.
*/
public const INDEX_COMPAT_VERSION = 'v3';
/** Matches Flarum's Search::MIN_SEARCH_LEN — the default minimum query length. */
public const DEFAULT_MIN_SEARCH_LENGTH = 3;
/** N-gram bounds for the title autocomplete subfield (not admin-configurable). */
private const TITLE_MIN_GRAM = 3;
private const TITLE_MAX_GRAM = 20;
protected $signature = 'blomstra:search:index
{--max-id= : Limits for each object the number of items to seed}
{--throttle= : Number of seconds to wait between pushing to the queue}
{--only= : type of model to index, eg discussions or posts}
{--recreate : create or recreate the index}
{--mapping : recreate the mapping}
{--continue : continue each object type where you left off}
{--seed-missing : attempt to seed only objects that are missing in the index}';
{action? : build | promote | rollback | discard | mapping | fill}
{--fresh}
{--resume}
{--staging}
{--keep-backup}
{--pending}
{--backup}
{--only=}
{--max-id=}
{--throttle=}
{--i-am-sure}';
protected $description = 'Rebuilds the complete search server with its documents.';
protected $description = 'Build and manage the Elasticsearch search index.';
public function handle(Container $container, Client $client, Queue $queue, SettingsRepositoryInterface $settings)
protected $help = <<<'HELP'
<comment>Actions and their options:</comment>
<info>build</info> Queue documents and promote automatically.
<comment>--resume</comment> Resume an interrupted build from where it left off
<comment>--fresh</comment> Drop the staging index and start completely fresh
<comment>--staging</comment> Keep in staging requires explicit <info>promote</info> (blue-green)
<comment>--keep-backup</comment> Retain the replaced index for rollback
<info>promote</info> Swap alias to staging index (blue-green workflow).
<comment>--keep-backup</comment> Retain the replaced index for rollback
<comment>--i-am-sure</comment> Skip the confirmation prompt
<info>rollback</info> Restore the backup index to live.
<info>discard</info> Drop an index without promoting.
<comment>--pending</comment> Drop the staging index (cancel a build)
<comment>--backup</comment> Drop the backup index (cleanup after --keep-backup)
<info>mapping</info> Push updated mapping to the live index only.
<info>fill</info> Seed only documents missing from the live index.
<comment>Shared seeding options (build / fill):</comment>
<comment>--only=TYPE</comment> Seed only this type: <info>discussions</info> or <info>posts</info>
<comment>--throttle=N</comment> Seconds to wait between batches
<comment>--max-id=N</comment> Limit seeding to IDs up to this value
HELP;
public function handle(Container $container): void
{
$indexers = [
'discussions' => DiscussionIndexer::class,
'posts' => CommentPostIndexer::class,
];
$models = [
'discussions' => Discussion::class,
'posts' => Post::class,
];
/** @var Client $client */
$client = $container->make(Client::class);
$only = $this->option('only');
$onlyIndexers = $only
? Arr::only($indexers, is_array($only) ? $only : explode(',', $only))
: $indexers;
/** @var SettingsRepositoryInterface $settings */
$settings = $container->make(SettingsRepositoryInterface::class);
foreach ($onlyIndexers as $type => $indexerClass) {
/** @var DiscussionIndexer|CommentPostIndexer $indexer */
$indexer = $container->make($indexerClass);
/** @var string $alias */
$alias = $container->make('blomstra.search.elastic_index');
$properties = [
'properties' => $indexer->properties(),
];
// Remove/delete the whole index.
if ($this->option('recreate')) {
$indexer->flush();
$indexer->build();
if (!$this->argument('action')) {
$this->call('help', ['command_name' => $this->getName()]);
return;
}
// Create the index.
if (!$this->option('recreate') && $this->option('mapping')) {
switch ($this->argument('action')) {
case 'build':
$this->runBuild($client, $alias, $settings, $container);
break;
case 'promote':
$this->runPromote($client, $alias, $settings);
break;
case 'rollback':
$this->runRollback($client, $alias, $settings);
break;
case 'discard':
$this->runDiscard($client, $settings);
break;
case 'mapping':
$client->indices()->putMapping([
'index' => $indexer::index(),
'body' => $properties,
'index' => $alias,
'body' => $this->mappingProperties(),
]);
$this->info('Mapping updated on live index.');
break;
case 'fill':
$this->runSeeders(
collect($container->tagged('blomstra.search.seeders')),
$container->make(Queue::class),
$client,
$settings,
$alias,
seedMissing: true
);
break;
default:
$this->error("Unknown action '{$this->argument('action')}'. Valid actions: build, promote, rollback, discard, mapping, fill.");
}
}
protected function runBuild(Client $client, string $alias, SettingsRepositoryInterface $settings, Container $container): void
{
/** @var Seeder[] $seeders */
$seeders = collect($container->tagged('blomstra.search.seeders'));
// Guard: staging build exists without explicit intent given.
$staging = $settings->get('blomstra-search.staging-index');
if ($staging && $client->indices()->exists(['index' => $staging])
&& !$this->option('resume') && !$this->option('fresh')
) {
$this->warn("An in-progress build already exists: $staging");
$this->line('');
$this->line('Choose one of:');
$this->line(' blomstra:search:index build --resume Continue from where it left off');
$this->line(' blomstra:search:index build --resume --staging Continue and keep in staging when done');
$this->line(' blomstra:search:index build --fresh Drop this build and start completely fresh');
$this->line(' blomstra:search:index build --fresh --staging Start fresh and keep in staging when done');
return;
}
$aliasExists = (bool) $client->indices()->existsAlias(['name' => $alias]);
$indexExists = !$aliasExists && (bool) $client->indices()->exists(['index' => $alias]);
if (!$aliasExists && !$indexExists) {
// First install: create a timestamped index, alias it immediately, seed live.
$targetIndex = $this->prepareFirstInstall($client, $alias, $settings, $seeders);
$stagingBuild = false;
} else {
// Live system: build into a staging index without touching the live alias.
$targetIndex = $this->prepareStagingIndex($client, $alias, $settings, $seeders);
$stagingBuild = true;
}
$client->indices()->putMapping([
'index' => $targetIndex,
'body' => $this->mappingProperties(),
]);
$this->runSeeders($seeders, $container->make(Queue::class), $client, $settings, $targetIndex);
if ($stagingBuild) {
if ($this->option('staging')) {
$staging = $settings->get('blomstra-search.staging-index');
$this->info("Staging build complete. Drain the queue, then promote '$staging' to live:");
$this->line(' php flarum queue:work --stop-when-empty');
$this->line(' php flarum blomstra:search:index promote');
} else {
$this->info('Jobs queued. Promoting now (search improves as the queue drains)...');
$this->performPromote($client, $alias, $settings);
}
}
}
protected function runPromote(Client $client, string $alias, SettingsRepositoryInterface $settings): void
{
if (!$this->option('i-am-sure')) {
$this->warn('promote switches the live index. Ensure all queued jobs have finished first:');
$this->line(' php flarum queue:work --stop-when-empty');
$this->line('');
if (!$this->confirm('Queue drained? Proceed?')) {
return;
}
}
$this->performPromote($client, $alias, $settings);
}
protected function performPromote(Client $client, string $alias, SettingsRepositoryInterface $settings): void
{
$staging = $settings->get('blomstra-search.staging-index');
if (!$staging || !$client->indices()->exists(['index' => $staging])) {
$this->error('No staging index ready to promote. Run: blomstra:search:index build');
return;
}
$aliasExists = (bool) $client->indices()->existsAlias(['name' => $alias]);
$indexExists = !$aliasExists && (bool) $client->indices()->exists(['index' => $alias]);
if ($aliasExists) {
$result = $client->indices()->getAlias(['name' => $alias]);
$activeIndex = array_key_first($result);
$client->indices()->updateAliases([
'body' => ['actions' => [
['remove' => ['index' => $activeIndex, 'alias' => $alias]],
['add' => ['index' => $staging, 'alias' => $alias]],
]],
]);
$this->info("'$alias' now points to '$staging'. Promote complete.");
if ($this->option('keep-backup')) {
$settings->set('blomstra-search.backup-index', $activeIndex);
$this->info("Kept '$activeIndex' as backup — run rollback to revert, discard --backup to drop it.");
} else {
$client->indices()->delete(['index' => $activeIndex, 'ignore_unavailable' => true]);
$this->info("Deleted old index: $activeIndex");
}
} else {
// One-time migration: concrete index → alias (legacy installs only).
if ($indexExists) {
$client->indices()->delete(['index' => $alias]);
$this->info("Deleted legacy concrete index: $alias");
}
$client->indices()->putAlias(['index' => $staging, 'name' => $alias]);
$this->info("'$alias' now points to '$staging'. Promote complete.");
}
$settings->set('blomstra-search.active-index', $staging);
$settings->set('blomstra-search.staging-index', null);
$this->saveIndexedConfig($client, $settings, $staging);
}
protected function runRollback(Client $client, string $alias, SettingsRepositoryInterface $settings): void
{
$backup = $settings->get('blomstra-search.backup-index');
if (!$backup || !$client->indices()->exists(['index' => $backup])) {
$this->error('No backup index to roll back to. Was promote run with --keep-backup?');
return;
}
$result = $client->indices()->getAlias(['name' => $alias]);
$activeIndex = array_key_first($result);
$client->indices()->updateAliases([
'body' => ['actions' => [
['remove' => ['index' => $activeIndex, 'alias' => $alias]],
['add' => ['index' => $backup, 'alias' => $alias]],
]],
]);
$client->indices()->delete(['index' => $activeIndex, 'ignore_unavailable' => true]);
$this->info("Rolled back: '$alias' now points to '$backup'. Deleted '$activeIndex'.");
$settings->set('blomstra-search.active-index', $backup);
$settings->set('blomstra-search.backup-index', null);
$this->saveIndexedConfig($client, $settings, $backup);
}
protected function runDiscard(Client $client, SettingsRepositoryInterface $settings): void
{
$pending = $this->option('pending');
$backup = $this->option('backup');
if (!$pending && !$backup) {
$this->error('Specify what to discard: --pending (staging build) or --backup (rollback index).');
return;
}
if ($pending) {
$staging = $settings->get('blomstra-search.staging-index');
if (!$staging) {
$this->warn('No staging index to discard.');
} else {
if ($client->indices()->exists(['index' => $staging])) {
$client->indices()->delete(['index' => $staging]);
}
$settings->set('blomstra-search.staging-index', null);
$this->info("Discarded staging index: $staging");
}
}
if ($backup) {
$backupIndex = $settings->get('blomstra-search.backup-index');
if (!$backupIndex) {
$this->warn('No backup index to discard.');
} else {
if ($client->indices()->exists(['index' => $backupIndex])) {
$client->indices()->delete(['index' => $backupIndex]);
}
$settings->set('blomstra-search.backup-index', null);
$this->info("Discarded backup index: $backupIndex");
}
}
}
protected function runSeeders(
iterable $seeders,
Queue $queue,
Client $client,
SettingsRepositoryInterface $settings,
string $targetIndex,
bool $seedMissing = false
): void {
$only = $this->option('only');
/** @var Seeder $seeder */
foreach ($seeders as $seeder) {
if ($only && $seeder->type() !== $only) {
continue;
}
$total = 0;
$query = $models[$type]::query();
if ($this->option('resume')) {
$saved = $this->getContinueAt($settings, $seeder->type());
$continueAt = $this->option('continue')
? ($this->continueAt($type) ?? $query->max('id'))
: $query->max('id');
if ($saved === 0) {
$this->info("Seeder '{$seeder->type()}' already completed in a previous run, skipping.");
continue;
}
$continueAt = $saved ?? $seeder->query()->max('id');
} else {
$continueAt = $seeder->query()->max('id');
}
$seeded = null;
while ($continueAt !== null) {
if ($this->option('seed-missing')) {
$response = (new Builder($client))
->index($indexer::index())
->size(1000)
->addQuery(
(new BoolQuery())
->add((new RangeQuery('rawId'))
->gte($continueAt - 1000)
->lte($continueAt))
)
->search();
$rangeFrom = max(1, $continueAt - 2500);
$rangeTo = $continueAt;
$seeded = Arr::pluck(Arr::get($response, 'hits.hits'), '_source.rawId');
if ($seedMissing) {
$seeded = $this->queryIndexedIds($client, $targetIndex, $seeder->joinRelation(), $rangeFrom, $rangeTo);
}
/** @var Collection $collection */
$collection = $query
$collection = $seeder->query()
->latest('id')
->whereBetween('id', [$continueAt - 1000, $continueAt])
->when($this->option('max-id'), function ($query, $id) {
$query->where('id', '<=', $id);
})
->when($seeded, fn ($query, $seeded) => $query->whereNotIn('id', $seeded))
->whereBetween('id', [$rangeFrom, $rangeTo])
->when($this->option('max-id'), fn ($q, $id) => $q->where('id', '<=', $id))
->when($seeded, fn ($q, $seeded) => $q->whereNotIn('id', $seeded))
->get();
$min = $collection->min('id');
if ($seedMissing && $collection->isEmpty()) {
$continueAt = $rangeFrom > 2 ? $rangeFrom - 1 : null;
} else {
$continueAt = $min && $min > 2 ? $min - 1 : null;
}
$onQueue = property_exists($indexerClass, 'queue') ? $indexerClass::$queue : null;
$queue->pushOn($onQueue, new IndexJob($indexerClass, $collection->all(), IndexJob::SAVE));
if ($collection->isNotEmpty()) {
$queue->pushOn(Job::$onQueue, new UpdateSearchJob($collection, $seeder, $targetIndex));
}
$this->info("Pushed into the index, type: $type, amount: {$collection->count()}.");
$this->info("IDs {$rangeFrom}{$rangeTo} | type: {$seeder->type()} | queued: {$collection->count()}.");
$total += $collection->count();
$this->continueAt($type, $continueAt);
$this->setContinueAt($settings, $seeder->type(), $continueAt);
if ($throttle = $this->option('throttle')) {
$this->info("Throttling for $throttle seconds");
@ -133,21 +400,274 @@ class BuildCommand extends Command
}
}
$this->info("Pushed a total of $total into the index.");
$this->setContinueAt($settings, $seeder->type(), 0);
$this->info("Queued a total of $total {$seeder->type()} for indexing.");
}
}
protected function continueAt(string $type, int $at = null)
/**
* First install: create a timestamped concrete index, alias the configured name to it
* immediately, and return the alias so seeding goes through it and is live from the start.
*/
protected function prepareFirstInstall(
Client $client,
string $alias,
SettingsRepositoryInterface $settings,
iterable $seeders
): string {
$concrete = $alias . '_' . date('YmdHis');
$client->indices()->create([
'index' => $concrete,
'body' => ['settings' => $this->buildIndexSettings($settings)],
]);
$client->indices()->putAlias(['index' => $concrete, 'name' => $alias]);
$settings->set('blomstra-search.active-index', $concrete);
$this->saveIndexedConfig($client, $settings, $concrete);
foreach ($seeders as $seeder) {
$this->setContinueAt($settings, $seeder->type(), null);
}
$this->info("Created '$concrete', aliased '$alias' → '$concrete'.");
$this->info("Index is live — documents become searchable as the queue processes.");
return $alias;
}
/**
* Prepare the staging index for a blue-green build.
*
* - If a staging build exists and --fresh is not set, resume it.
* - Otherwise create a fresh timestamped index and save it as staging.
*/
protected function prepareStagingIndex(
Client $client,
string $alias,
SettingsRepositoryInterface $settings,
iterable $seeders
): string {
$staging = $settings->get('blomstra-search.staging-index');
if ($this->option('fresh') && $staging) {
if ($client->indices()->exists(['index' => $staging])) {
$client->indices()->delete(['index' => $staging]);
$this->info("Dropped staging index: $staging");
}
$staging = null;
$settings->set('blomstra-search.staging-index', null);
}
if ($staging && $client->indices()->exists(['index' => $staging])) {
$this->info("Resuming staging index build: $staging");
return $staging;
}
$staging = $alias . '_' . date('YmdHis');
$client->indices()->create([
'index' => $staging,
'body' => ['settings' => $this->buildIndexSettings($settings)],
]);
$settings->set('blomstra-search.staging-index', $staging);
foreach ($seeders as $seeder) {
$this->setContinueAt($settings, $seeder->type(), null);
}
$this->info("Created staging index: $staging");
return $staging;
}
/**
* Persist the analysis config and compat version that are actually live in ES for the
* given index. Reading from ES (rather than from Flarum settings) means rollbacks are
* also covered: the stored values always reflect the index that is currently aliased,
* not the settings at the time the command ran. Indexes built before _meta tracking
* existed will yield a null compat version, which correctly triggers the reindex warning.
*/
protected function saveIndexedConfig(Client $client, SettingsRepositoryInterface $settings, string $indexName): void
{
/** @var SettingsRepositoryInterface $settings */
$settings = resolve(SettingsRepositoryInterface::class);
$settingsResponse = $client->indices()->getSettings(['index' => $indexName]);
$analysis = Arr::get($settingsResponse, "$indexName.settings.index.analysis", []);
$key = "blomstra-search.continued-at.$type";
$analyzer = Arr::get($analysis, 'analyzer.flarum_analyzer.type', 'english');
$stemExclusion = Arr::get($analysis, 'analyzer.flarum_analyzer.stem_exclusion', []);
if ($at) {
$settings->set($key, $at);
} else {
return $settings->get($key);
$mappingResponse = $client->indices()->getMapping(['index' => $indexName]);
$compatVersion = Arr::get($mappingResponse, "$indexName.mappings._meta.index_compat_version");
$settings->set('blomstra-search.indexed-analyzer', $analyzer);
$settings->set('blomstra-search.indexed-stem-exclusion', implode("\n", $stemExclusion));
$settings->set('blomstra-search.index-compatible', $compatVersion);
}
protected function buildIndexSettings(SettingsRepositoryInterface $settings): array
{
$language = $settings->get('blomstra-search.analyzer-language') ?: 'english';
$raw = $settings->get('blomstra-search.stem-exclusion', '');
$stemExclusion = array_values(array_filter(array_map('trim', explode("\n", $raw))));
if ($language === 'cjk') {
// CJK uses the built-in bigram analyzer; no autocomplete subfield.
return [
'analysis' => [
'analyzer' => [
'flarum_analyzer' => ['type' => 'cjk'],
],
],
];
}
$analyzerConfig = ['type' => $language];
if (!empty($stemExclusion)) {
$analyzerConfig['stem_exclusion'] = $stemExclusion;
}
return [
'index.max_ngram_diff' => self::TITLE_MAX_GRAM - self::TITLE_MIN_GRAM,
'analysis' => [
'filter' => [
'title_autocomplete_filter' => [
'type' => 'edge_ngram',
'min_gram' => self::TITLE_MIN_GRAM,
'max_gram' => self::TITLE_MAX_GRAM,
'token_chars' => ['letter', 'digit'],
],
],
'analyzer' => [
// Symmetric language analyzer: used for BOTH index and search on content + title.
'flarum_analyzer' => $analyzerConfig,
// Title autocomplete index-time: prefix n-grams, no stemming.
'flarum_title_autocomplete' => [
'type' => 'custom',
'tokenizer' => 'standard',
'filter' => ['lowercase', 'title_autocomplete_filter'],
],
// Title autocomplete search-time: just lowercase — predictable on partial words.
'flarum_title_search' => [
'type' => 'custom',
'tokenizer' => 'standard',
'filter' => ['lowercase'],
],
],
],
];
}
protected function mappingProperties(): array
{
$language = resolve(SettingsRepositoryInterface::class)->get('blomstra-search.analyzer-language') ?: 'english';
$titleMapping = [
'type' => 'text',
'analyzer' => 'flarum_analyzer',
];
// Only add the autocomplete subfield for non-CJK languages.
if ($language !== 'cjk') {
$titleMapping['fields'] = [
'autocomplete' => [
'type' => 'text',
'analyzer' => 'flarum_title_autocomplete',
'search_analyzer' => 'flarum_title_search',
],
];
}
$contentMapping = [
'type' => 'text',
'analyzer' => 'flarum_analyzer',
];
// Only add the autocomplete subfield for non-CJK languages.
if ($language !== 'cjk') {
$contentMapping['fields'] = [
'autocomplete' => [
'type' => 'text',
'analyzer' => 'flarum_title_autocomplete',
'search_analyzer' => 'flarum_title_search',
],
];
}
return [
'_meta' => ['index_compat_version' => self::INDEX_COMPAT_VERSION],
'properties' => [
'join_field' => ['type' => 'join', 'relations' => ['discussion' => 'post']],
'discussion_id' => ['type' => 'integer'],
'content' => $contentMapping,
'title' => $titleMapping,
'rawId' => ['type' => 'integer'],
'created_at' => ['type' => 'date'],
'updated_at' => ['type' => 'date'],
'is_private' => ['type' => 'boolean'],
'user_id' => ['type' => 'integer'],
'groups' => ['type' => 'keyword'],
'comment_count' => ['type' => 'integer'],
'tags' => ['type' => 'keyword'],
'is_sticky' => ['type' => 'boolean'],
'is_hidden' => ['type' => 'boolean'],
],
];
}
/**
* Query ES for rawIds already indexed in $targetIndex for the given $joinRelation and ID range.
* Retries on transient ES failures (NoNodesAvailableException / other ElasticsearchException)
* by sleeping past the StaticNoPingConnectionPool dead-node timeout (default 60 s) before
* each retry, giving the pool a chance to resurface the node.
*/
protected function queryIndexedIds(
Client $client,
string $targetIndex,
string $joinRelation,
int $rangeFrom,
int $rangeTo,
int $maxRetries = 10
): array {
$attempt = 0;
while (true) {
try {
$response = (new Builder($client))
->index($targetIndex)
->size(2500)
->addQuery(
(new BoolQuery())
->add((new RangeQuery('rawId'))->gte($rangeFrom)->lte($rangeTo))
->add(TermQuery::create('join_field', $joinRelation))
)
->search();
return Arr::pluck(Arr::get($response, 'hits.hits'), '_source.rawId');
} catch (ElasticsearchException $e) {
$attempt++;
if ($attempt >= $maxRetries) {
throw $e;
}
$this->warn("ES error on range {$rangeFrom}{$rangeTo} (attempt {$attempt}/{$maxRetries}): {$e->getMessage()}. Waiting 65 s before retry…");
sleep(65); // outlast the StaticNoPingConnectionPool dead-node window (default 60 s)
}
}
}
protected function getContinueAt(SettingsRepositoryInterface $settings, string $type): ?int
{
$raw = $settings->get("blomstra-search.continued-at.$type");
return $raw !== null ? (int) $raw : null;
}
protected function setContinueAt(SettingsRepositoryInterface $settings, string $type, ?int $at): void
{
$settings->set("blomstra-search.continued-at.$type", $at);
}
}

View File

@ -1,111 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Discussion;
use Blomstra\Search\Save\Document;
use Blomstra\Search\Search\Concerns\AppliesAccessControl;
use Blomstra\Search\Search\ElasticIndex;
use Flarum\Discussion\Discussion;
use Flarum\Extension\ExtensionManager;
use Flarum\Search\IndexerInterface;
class DiscussionIndexer implements IndexerInterface
{
use AppliesAccessControl;
public function __construct(
protected ElasticIndex $elastic,
protected ExtensionManager $extensions
) {
}
public static function index(): string
{
return 'discussions';
}
public function save(array $models): void
{
$this->elastic->save(self::index(), $models, $this->toDocument(...));
}
public function delete(array $models): void
{
$this->elastic->delete(self::index(), $models);
}
public function build(): void
{
$this->elastic->build(self::index(), $this->properties());
}
public function flush(): void
{
$this->elastic->flush(self::index());
}
public function properties(): array
{
return [
'rawId' => ['type' => 'integer'],
'discussion_id' => ['type' => 'integer'],
'title' => ['type' => 'text', 'analyzer' => 'flarum_analyzer_partial', 'search_analyzer' => 'flarum_analyzer'],
'created_at' => ['type' => 'date'],
'updated_at' => ['type' => 'date'],
'last_posted_at' => ['type' => 'date'],
'is_private' => ['type' => 'boolean'],
'is_sticky' => ['type' => 'boolean'],
'is_locked' => ['type' => 'boolean'],
'groups' => ['type' => 'integer'],
'tags' => ['type' => 'integer'],
'recipient_groups' => ['type' => 'integer'],
'recipient_users' => ['type' => 'integer'],
'comment_count' => ['type' => 'integer'],
];
}
public function toDocument(Discussion $model): Document
{
$document = new Document([
'id' => $model->id,
'discussion_id' => $model->id, // duplicated for result aggregation in searching with posts.
'rawId' => $model->id,
'title' => $model->title,
'created_at' => $model->created_at?->toAtomString(),
'updated_at' => $model->last_posted_at?->toAtomString(),
'is_private' => $model->is_private,
'user_id' => $model->user_id,
'groups' => $this->groupsForDiscussion($model),
'comment_count' => $model->comment_count,
]);
if ($this->extensions->isEnabled('flarum-tags')) {
$document['tags'] = $model->tags->pluck('id')->toArray();
}
if ($this->extensions->isEnabled('fof-byobu')) {
$document['recipient_users'] = $model->recipientUsers->pluck('id')->toArray();
$document['recipient_groups'] = $model->recipientGroups->pluck('id')->toArray();
}
if ($this->extensions->isEnabled('flarum-sticky')) {
$document['is_sticky'] = $model->is_sticky;
}
if ($this->extensions->isEnabled('flarum-lock')) {
$document['is_locked'] = $model->is_locked;
}
return $document;
}
}

View File

@ -1,32 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Discussion;
use Blomstra\Search\Post\CommentPostIndexer;
use Blomstra\Search\Search\Searcher;
use Flarum\Discussion\Discussion;
use Flarum\User\User;
use Illuminate\Database\Eloquent\Builder;
class DiscussionSearcher extends Searcher
{
public function index(): string
{
return DiscussionIndexer::index().','.CommentPostIndexer::index();
}
public function getQuery(User $actor): Builder
{
return Discussion::whereVisibleTo($actor)->select('discussions.*');
}
}

View File

@ -1,166 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Discussion;
use Blomstra\Search\Elasticsearch\Builder;
use Blomstra\Search\Elasticsearch\MatchPhraseQuery;
use Blomstra\Search\Elasticsearch\MatchQuery;
use Blomstra\Search\Elasticsearch\TermsQuery;
use Blomstra\Search\Post\CommentPostIndexer;
use Blomstra\Search\Search\ElasticSearchState;
use Elasticsearch\Client;
use Flarum\Discussion\Discussion;
use Flarum\Post\Post;
use Flarum\Search\AbstractFulltextFilter;
use Flarum\Search\SearchCriteria;
use Flarum\Search\SearchManager;
use Flarum\Search\SearchState;
use Illuminate\Support\Arr;
use Illuminate\Support\Collection;
use Spatie\ElasticsearchQueryBuilder\Aggregations\FilterAggregation;
use Spatie\ElasticsearchQueryBuilder\Aggregations\TermsAggregation;
use Spatie\ElasticsearchQueryBuilder\Aggregations\TopHitsAggregation;
use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
use Spatie\ElasticsearchQueryBuilder\Sorts\Sort;
/**
* Unlike the default flarum database search,
* this search will not get score based on the sum of post scores of a discussion.
*
* @extends AbstractFulltextFilter<ElasticSearchState>
*/
class FulltextFilter extends AbstractFulltextFilter
{
public function __construct(
protected SearchManager $search,
protected Client $elastic
) {
}
public function search(SearchState $state, string $value): void
{
$builder = $state->getBuilder();
$query = BoolQuery::create()
->add(
BoolQuery::create()
->add($this->exactMatch('title', $value, 1.2), 'should')
->add($this->wordMatch('title', $value, 'and', 1.2), 'should')
->add($this->wordMatch('title', $value, 'or', 1.2), 'should'),
'should'
)
->add(
BoolQuery::create()
->add($this->exactMatch('content', $value), 'should', 1)
->add($this->wordMatch('content', $value, 'and', 1), 'should')
->add($this->wordMatch('content', $value, 'or', 1), 'should'),
'should'
);
$builder->addQuery($query);
$builder->collapse('discussion_id');
$aggs = FilterAggregation::create('posts', TermsQuery::create('_index', [CommentPostIndexer::index()]))
->aggregation(
TermsAggregation::create('per_discussion', 'discussion_id')
->aggregation(
TopHitsAggregation::create('most_relevant_post_id', 1, Sort::create('_score'))
)
);
$builder->addAggregation($aggs);
$state->setDefaultSort(function (Builder $builder) {
$builder->addSort(
Sort::create('_score', 'desc')
);
});
$state->retrieveDatabaseRecordsUsing(function (array $response, SearchCriteria $criteria): Collection {
$buckets = Collection::make(Arr::get($response, 'aggregations.posts.per_discussion.buckets'))
->map(fn (array $hit) => [
'discussion_id' => $hit['key'],
'most_relevant_post_id' => Arr::get($hit, 'most_relevant_post_id.most_relevant_post_id.hits.hits.0._id'),
'most_relevant_post_score' => Arr::get($hit, 'most_relevant_post_id.most_relevant_post_id.hits.hits.0._score'),
])->keyBy('discussion_id');
$results = Collection::make(Arr::get($response, 'hits.hits'))
->map(fn (array $hit) => [
'discussion_id' => $hit['_source']['discussion_id'],
'most_relevant_post_id' => Arr::get($buckets->get($hit['_source']['discussion_id']), 'most_relevant_post_id'),
'most_relevant_post_score' => Arr::get($buckets->get($hit['_source']['discussion_id']), 'most_relevant_post_score'),
'title_score' => $hit['_score'],
])->keyBy('discussion_id');
// We have $hits and $buckets, both are sorted by score.
// $discussionResult represents discussion scores based on the title.
// $buckets represents discussion scores based on the sum of all posts,
// and also the most relevant post id and its score.
// We need to merge these two results into one collection, and sort them by score.
// We also need to make sure that the most relevant post id is set on the discussion.
$results = $results
// Sort by title_score, most_relevant_post_score then posts_score.
->sortByDesc(function ($result) {
return $result['title_score'] ?? 0
+ $result['most_relevant_post_score'] ?? 0
+ $result['posts_score'] ?? 0;
})
->take($criteria->limit);
$actor = $criteria->actor;
$connection = Post::query()->getConnection();
if ($results->isEmpty()) {
return Discussion::whereVisibleTo($actor)
->select('discussions.*')
->get();
}
return Discussion::whereVisibleTo($actor)
->select('discussions.*')
->selectRaw(
$connection->raw('COALESCE(('.$connection->getQueryGrammar()->compileSelect(
$postsQuery = Post::query()
->select('posts.id')
->whereIn('posts.id', $results->pluck('most_relevant_post_id')->filter())
->whereColumn('discussions.id', '=', 'posts.discussion_id')
->limit(1)
->toBase()
).'), first_post_id) as most_relevant_post_id')->getValue($connection->getQueryGrammar())
)
->mergeBindings($postsQuery)
->whereIn('discussions.id', $results->pluck('discussion_id'))
->orderByRaw('FIELD(`discussions`.`id`, '.implode(',', $results->pluck('discussion_id')->all()).')')
->get();
});
}
protected function exactMatch(string $field, string $q, int $fieldBoost = 1): Query
{
$query = (new MatchPhraseQuery($field, $q));
return $query->boost(2 * $fieldBoost);
}
protected function wordMatch(string $field, string $q, string $operator, int $fieldBoost = 1): Query
{
$query = (new MatchQuery($field, $q))
->operator($operator);
$boost = $operator === 'and' ? 1.8 : .8;
return $query->boost($boost * $fieldBoost);
}
}

View File

@ -1,95 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Discussion;
use Blomstra\Search\Elasticsearch\TermsQuery;
use Blomstra\Search\Search\Concerns\AppliesAccessControl;
use Blomstra\Search\Search\ElasticSearchState;
use Flarum\Extension\ExtensionManager;
use Flarum\Search\Filter\FilterInterface;
use Flarum\Search\SearchState;
use Illuminate\Support\Arr;
use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery;
/**
* @implements FilterInterface<ElasticSearchState>
*/
class PrivateFilterMutator implements FilterInterface
{
use AppliesAccessControl;
public function __construct(
protected ExtensionManager $extensions
) {
}
public function getFilterKey(): string
{
return 'private';
}
public function filter(SearchState $state, array|string $value, bool $negate): void
{
$actor = $state->getActor();
if (!$this->extensions->isEnabled('fof-byobu') || $actor->isGuest()) {
return;
}
$builder = $state->getBuilder();
$query = BoolQuery::create();
$query->add(self::byobuAccessQuery($state), 'filter');
$builder->addQuery($query);
}
public static function mutate(ElasticSearchState $state): void
{
$builder = $state->getBuilder();
// If this filter isn't active, we apply both the private and public queries as should clauses.
if (Arr::first($state->getActiveFilters(), fn ($filter) => $filter->getFilterKey() === 'private')) {
return;
}
$query = BoolQuery::create();
$query
->add(self::restrictQuery($state), 'should')
->add(self::byobuAccessQuery($state), 'should');
$builder->addQuery($query);
}
private static function restrictQuery(SearchState $state): BoolQuery
{
return BoolQuery::create()
->add(TermQuery::create('is_private', 'false'))
->add(TermsQuery::create('groups', self::groupsForUser($state->getActor())));
}
private static function byobuAccessQuery(SearchState $state): BoolQuery
{
$actor = $state->getActor();
return BoolQuery::create()
->add(TermQuery::create('is_private', 'true'))
->add(
BoolQuery::create()
->add(TermsQuery::create('recipient_groups', self::groupsForUser($actor)), 'should')
->add(TermsQuery::create('recipient_users', [$actor->id]), 'should'),
);
}
}

View File

@ -0,0 +1,41 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Elasticsearch;
class BoolQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery
{
protected ?int $minimumShouldMatch = null;
public static function create(): static
{
return new self();
}
public function minimumShouldMatch(int $minimum): static
{
$this->minimumShouldMatch = $minimum;
return $this;
}
public function toArray(): array
{
$array = parent::toArray();
if ($this->minimumShouldMatch !== null) {
$array['bool']['minimum_should_match'] = $this->minimumShouldMatch;
}
return $array;
}
}

View File

@ -1,68 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch;
class Builder extends \Spatie\ElasticsearchQueryBuilder\Builder
{
protected array $collapse = [];
public function collapse(string $field): static
{
$this->collapse = [
'field' => $field,
];
return $this;
}
public function getPayload(): array
{
$payload = parent::getPayload();
if ($this->collapse) {
$payload['collapse'] = $this->collapse;
}
return $payload;
}
public function search(): array
{
$params = $this->getParams();
return $this->client->search($params);
}
public function getParams(): array
{
$payload = $this->getPayload();
$params = [
'body' => $payload,
];
if ($this->searchIndex) {
$params['index'] = $this->searchIndex;
}
if ($this->size !== null) {
$params['size'] = $this->size;
}
if ($this->from !== null) {
$params['from'] = $this->from;
}
return $params;
}
}

View File

@ -0,0 +1,57 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
class HasChildQuery implements Query
{
protected bool $innerHits = false;
public function __construct(
protected string $type,
protected Query $query,
protected string $scoreMode = 'sum'
) {}
public static function create(string $type, Query $query, string $scoreMode = 'sum'): static
{
return new static($type, $query, $scoreMode);
}
/**
* Include inner_hits so the best-matching post ID is available in the response.
* ES returns the top-scoring child document per parent hit under inner_hits.best_post.
*/
public function withInnerHits(): static
{
$this->innerHits = true;
return $this;
}
public function toArray(): array
{
$hasChild = [
'type' => $this->type,
'score_mode' => $this->scoreMode,
'query' => $this->query->toArray(),
];
if ($this->innerHits) {
$hasChild['inner_hits'] = ['name' => 'best_post', 'size' => 1];
}
return ['has_child' => $hasChild];
}
}

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Elasticsearch;
namespace Sbp\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery;

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Elasticsearch;
namespace Sbp\Search\Elasticsearch;
class MatchQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery
{
@ -62,6 +62,10 @@ class MatchQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery
$query['match'][$this->field]['boost'] = $this->boost;
$query['match'][$this->field]['zero_terms_query'] = $this->zeroTerms ? 'all' : 'none';
if ($this->analyzer !== null) {
$query['match'][$this->field]['analyzer'] = $this->analyzer;
}
return $query;
}
}

View File

@ -0,0 +1,59 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
class SimpleSearchQuery implements Query
{
protected float $boost = 1;
protected ?string $analyzer = null;
public static function create(array $field, string $value)
{
return new self($field, $value);
}
public function boost(float $boost = 1)
{
$this->boost = $boost;
return $this;
}
public function analyzer(string $analyzer)
{
$this->analyzer = $analyzer;
return $this;
}
public function __construct(
protected array $fields,
protected string $value
) {
}
public function toArray(): array
{
return [
'simple_query_string' => [
'query' => $this->value,
'fields' => $this->fields,
'analyzer' => $this->analyzer,
'default_operator' => 'AND',
'boost' => $this->boost,
],
];
}
}

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Elasticsearch;
namespace Sbp\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Elasticsearch;
namespace Sbp\Search\Elasticsearch;
class WildcardQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\WildcardQuery
{

View File

@ -10,13 +10,13 @@
*
*/
namespace Blomstra\Search\Exceptions;
namespace Sbp\Search\Exceptions;
use Throwable;
class IndexingException extends \Exception
class SeedingException extends \Exception
{
public function __construct(string $message, public array $items, $code = 0, Throwable $previous = null)
public function __construct($message = '', public array $items, $code = 0, Throwable $previous = null)
{
parent::__construct($message, $code, $previous);
}

42
src/Jobs/DeletingJob.php Normal file
View File

@ -0,0 +1,42 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Jobs;
use Elasticsearch\Client;
use Illuminate\Database\Eloquent\Model;
class DeletingJob extends Job
{
public function handle(Client $client)
{
if ($this->models->isEmpty()) {
return;
}
// Preparing body for storing.
$body = $this->models->map(function (Model $model) {
$document = $this->seeder->toDocument($model);
$routing = $this->seeder->routing($model);
return [
['delete' => ['_index' => $this->index, '_id' => $document->id, 'routing' => $routing]],
];
})->flatten(1);
$response = $client->bulk([
'index' => $this->index,
'body' => $body->toArray(),
'refresh' => true,
]);
}
}

37
src/Jobs/Job.php Normal file
View File

@ -0,0 +1,37 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Jobs;
use Sbp\Search\Seeders\Seeder;
use Flarum\Queue\AbstractJob;
use Illuminate\Database\Eloquent\Collection;
abstract class Job extends AbstractJob
{
protected string $index;
public static ?string $onQueue = null;
/**
* @param string|null $targetIndex Explicit index name for blue-green builds.
* Defaults to the configured alias when null.
*/
public function __construct(protected Collection $models, protected Seeder $seeder, ?string $targetIndex = null)
{
$this->index = $targetIndex ?? resolve('blomstra.search.elastic_index');
if (static::$onQueue) {
$this->onQueue(static::$onQueue);
}
}
}

View File

@ -0,0 +1,62 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Jobs;
use Sbp\Search\Exceptions\SeedingException;
use Elasticsearch\Client;
use Illuminate\Database\Eloquent\Collection as EloquentCollection;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Support\Arr;
class UpdateSearchJob extends Job
{
public function handle(Client $client)
{
if ($this->models->isEmpty()) {
return;
}
$this->models->loadMissing($this->seeder->relationships());
// Preparing body for storing.
$body = $this->models->map(function (Model $model) {
$document = $this->seeder->toDocument($model);
$routing = $this->seeder->routing($model);
return [
['index' => ['_index' => $this->index, '_id' => $document->id, 'routing' => $routing]],
$document->toArray(),
];
})
->flatten(1);
$response = $client->bulk([
'index' => $this->index,
'body' => $body->toArray(),
]);
if (Arr::get($response, 'errors') !== true) {
return true;
}
$items = Arr::get($response, 'items');
$failed = array_filter($items, fn ($item) => isset($item['index']['error']));
$error = Arr::get(Arr::first($failed), 'index.error.reason', 'unknown error');
throw new SeedingException(
"Failed to seed: $error (" . count($failed) . '/' . count($items) . ' items failed)',
$items
);
}
}

View File

@ -0,0 +1,59 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Jobs;
use Elasticsearch\Client;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Discussion\Discussion;
use Flarum\Queue\AbstractJob;
class ViewsSearchJob extends AbstractJob
{
protected string $index;
protected string $documentType;
public function __construct(protected int $discussionId)
{
$this->index = resolve('blomstra.search.elastic_index');
$this->documentType = resolve(DiscussionSerializer::class)->getType(new Discussion());
if (Job::$onQueue) {
$this->onQueue(Job::$onQueue);
}
}
public function handle(Client $client): void
{
$discussion = Discussion::find($this->discussionId);
if (!$discussion) {
return;
}
$type = $this->documentType;
try {
$client->update([
'index' => $this->index,
'id' => "$type:{$this->discussionId}",
'routing' => (string) $this->discussionId,
'retry_on_conflict' => 3,
'body' => [
'doc' => ['view_count' => (int) $discussion->view_count],
],
]);
} catch (\Elasticsearch\Common\Exceptions\Missing404Exception $e) {
// Document not yet indexed; will be picked up on next: blomstra:search:index fill
}
}
}

View File

@ -1,109 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Post;
use Blomstra\Search\Save\Document;
use Blomstra\Search\Search\Concerns\AppliesAccessControl;
use Blomstra\Search\Search\ElasticIndex;
use Flarum\Extension\ExtensionManager;
use Flarum\Post\CommentPost;
use Flarum\Post\Post;
use Flarum\Search\IndexerInterface;
class CommentPostIndexer implements IndexerInterface
{
use AppliesAccessControl;
public function __construct(
protected ElasticIndex $elastic,
protected ExtensionManager $extensions
) {
}
public static function index(): string
{
return 'posts';
}
public function save(array $models): void
{
$this->elastic->save(
self::index(),
array_filter($models, fn (Post $model) => $model->type === CommentPost::$type),
$this->toDocument(...)
);
}
public function delete(array $models): void
{
$this->elastic->delete(
self::index(),
array_filter($models, fn (Post $model) => $model->type === CommentPost::$type)
);
}
public function build(): void
{
$this->elastic->build(self::index(), $this->properties());
}
public function flush(): void
{
$this->elastic->flush(self::index());
}
public function properties(): array
{
return [
'rawId' => ['type' => 'integer'],
'content' => ['type' => 'text', 'analyzer' => 'flarum_analyzer_partial', 'search_analyzer' => 'flarum_analyzer'],
'discussion_id' => ['type' => 'integer'],
'created_at' => ['type' => 'date'],
'updated_at' => ['type' => 'date'],
'is_private' => ['type' => 'boolean'],
'groups' => ['type' => 'integer'],
'tags' => ['type' => 'integer'],
'recipient_groups' => ['type' => 'integer'],
'recipient_users' => ['type' => 'integer'],
'comment_count' => ['type' => 'integer'],
];
}
public function toDocument(Post $model): Document
{
$document = new Document([
'id' => $model->id,
'type' => $model->type,
'rawId' => $model->id,
'discussion_id' => $model->discussion_id,
'content' => $model->content,
'created_at' => $model->created_at?->toAtomString(),
'updated_at' => $model->edited_at?->toAtomString(),
'is_private' => $model->is_private,
'user_id' => $model->user_id,
'groups' => $this->groupsForDiscussion($model->discussion),
'comment_count' => $model->discussion->comment_count,
]);
if ($this->extensions->isEnabled('flarum-tags')) {
$document['tags'] = $model->discussion->tags->pluck('id')->toArray();
}
if ($this->extensions->isEnabled('fof-byobu')) {
$document['recipient_users'] = $model->discussion->recipientUsers->pluck('id')->toArray();
$document['recipient_groups'] = $model->discussion->recipientGroups->pluck('id')->toArray();
}
return $document;
}
}

View File

@ -10,29 +10,50 @@
*
*/
namespace Blomstra\Search;
namespace Sbp\Search;
use Sbp\Search\Jobs\DeletingJob;
use Sbp\Search\Jobs\Job;
use Sbp\Search\Jobs\UpdateSearchJob;
use Sbp\Search\Jobs\ViewsSearchJob;
use Elasticsearch\Client as Elastic;
use Elasticsearch\ClientBuilder;
use Flarum\Api\Client;
use Flarum\Foundation\AbstractServiceProvider;
use Flarum\Foundation\Config;
use Flarum\Http\Middleware\ExecuteRoute;
use Flarum\Settings\SettingsRepositoryInterface;
use Illuminate\Contracts\Container\Container;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Contracts\Queue\Queue;
use Illuminate\Database\Eloquent\Collection;
use Laminas\Stratigility\MiddlewarePipe;
use Psr\Log\LoggerInterface;
class Provider extends AbstractServiceProvider
{
public function register(): void
public function register()
{
$this->container->singleton(Elastic::class, function (Container $container) {
/** @var Config $config */
$config = $this->container->make(Config::class);
$this->container->tag([
Seeders\DiscussionSeeder::class,
Seeders\CommentSeeder::class,
], 'blomstra.search.seeders');
/** @var SettingsRepositoryInterface $settings */
$settings = $this->container->make(SettingsRepositoryInterface::class);
/** @var Config $config */
$config = $this->container->make(Config::class);
$this->container->singleton(Elastic::class, function (Container $container) use ($settings, $config) {
$builder = ClientBuilder::create()
->setHosts([$settings->get('blomstra-search.elastic-endpoint')]);
->setHosts([$settings->get('blomstra-search.elastic-endpoint')])
->setConnectionParams([
'client' => [
'connect_timeout' => 2, // fail fast if ES is unreachable
'timeout' => 10, // allow time for complex queries
],
]);
if ($config->inDebugMode()) {
$builder->setLogger($container->make(LoggerInterface::class));
@ -47,10 +68,63 @@ class Provider extends AbstractServiceProvider
return $builder->build();
});
$this->container->instance(
'blomstra.search.elastic_index',
$settings->get('blomstra-search.elastic-index', 'flarum')
);
$this->container->extend(
Client::class,
function () {
$pipe = new MiddlewarePipe();
$exclude = resolve('flarum.api_client.exclude_middleware');
$middlewareStack = array_filter(resolve('flarum.api.middleware'), function ($middlewareClass) use ($exclude) {
return !in_array($middlewareClass, $exclude);
});
foreach ($middlewareStack as $middleware) {
$pipe->pipe(resolve($middleware));
}
public function boot(): void
$pipe->pipe(new ExecuteRoute());
return new Api\Client($pipe);
}
);
$this->container->tag([
Searchers\DiscussionSearcher::class,
Searchers\CommentPostSearcher::class,
], 'blomstra.search.searchers');
}
public function boot()
{
//
/** @var array|string[] $seeders */
$seeders = $this->container->tagged('blomstra.search.seeders');
/** @var Dispatcher $events */
$events = resolve(Dispatcher::class);
/** @var Queue $queue */
$queue = resolve(Queue::class);
/** @var string|Seeders\Seeder $seeder */
foreach ($seeders as $seeder) {
$seeder::savingOn($events, function ($model) use ($queue, $seeder) {
$queue->pushOn(Job::$onQueue, new UpdateSearchJob(Collection::make([$model]), $seeder));
});
$seeder::deletingOn($events, function ($model) use ($queue, $seeder) {
$queue->pushOn(Job::$onQueue, new DeletingJob(Collection::make([$model]), $seeder));
});
$seeder::viewingOn($events, function (int $discussionId) use ($queue) {
$queue->pushOn(Job::$onQueue, new ViewsSearchJob($discussionId));
});
}
}
}

View File

@ -10,13 +10,15 @@
*
*/
namespace Blomstra\Search\Save;
namespace Sbp\Search\Save;
use Carbon\Carbon;
use Illuminate\Support\Arr;
use Illuminate\Support\Fluent;
/**
* @property string $id
* @property int $rawId
* @property string $content
* @property Carbon $created_at
* @property Carbon $updated_at
@ -29,4 +31,13 @@ use Illuminate\Support\Fluent;
*/
class Document extends Fluent
{
/**
* Exclude `id` from the body sent to Elasticsearch. The document ID is
* already stored as `_id` by the bulk API action; duplicating it in
* `_source` wastes space and is redundant.
*/
public function toArray(): array
{
return Arr::except($this->attributes, ['id']);
}
}

View File

@ -1,74 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Search\Concerns;
use Flarum\Discussion\Discussion;
use Flarum\Extension\ExtensionManager;
use Flarum\Group\Group;
use Flarum\Group\Permission;
use Flarum\Tags\Tag;
use Flarum\User\User;
trait AppliesAccessControl
{
protected static function groupsForUser(User $actor): array
{
$groups = $actor->groups->pluck('id');
$groups->add(Group::GUEST_ID);
if ($actor->is_email_confirmed) {
$groups->add(Group::MEMBER_ID);
}
return $groups->toArray();
}
protected function groupsForDiscussion(Discussion $discussion): array
{
$permissions = collect();
$globalPermission = Permission::query()
->where('permission', 'viewForum')
->pluck('group_id');
if (resolve(ExtensionManager::class)->isEnabled('flarum-tags')) {
/** @var \Illuminate\Database\Eloquent\Collection $tags */
$tags = $discussion->tags;
$tagPermissions = Permission::query()
->whereIn(
'permission',
$tags->pluck('id')->map(function (int $id) {
return "tag$id.viewForum";
})
)->get();
$permissions = $tags->map(function (Tag $tag) use ($tagPermissions) {
$permissions = $tagPermissions->where('permission', "tag$tag->id.viewForum");
if ($tag->is_restricted) {
$permissions = $permissions->add(['group_id' => Group::ADMINISTRATOR_ID]);
}
return $permissions->pluck('group_id');
})->flatten();
}
if (!$discussion->is_private && $permissions->isEmpty()) {
$permissions = $globalPermission;
}
return $permissions->toArray();
}
}

View File

@ -1,144 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Search;
use Blomstra\Search\Exceptions\IndexingException;
use Closure;
use Elasticsearch\Client;
use Flarum\Settings\SettingsRepositoryInterface;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Support\Arr;
class ElasticIndex
{
public function __construct(
protected Client $client,
protected SettingsRepositoryInterface $settings
) {
}
public function save(string $index, array $models, Closure $toDocument): void
{
$models = collect($models);
// Preparing body for storing.
$body = $models->map(function (Model $model) use ($index, $toDocument) {
$document = $toDocument($model);
return [
[
'index' => [
'_index' => $index,
'_id' => $document->id,
],
],
$document->toArray(),
];
})
->flatten(1);
$this->handleResponse(
$this->client->bulk([
'index' => $index,
'body' => $body->toArray(),
'refresh' => true,
])
);
}
public function delete(string $index, array $models): void
{
$models = collect($models);
$body = $models->map(function (Model $model) use ($index) {
return [
[
'delete' => [
'_index' => $index,
'_id' => $model->id,
],
],
];
})->flatten(1);
$this->handleResponse(
$this->client->bulk([
'index' => $index,
'body' => $body->toArray(),
'refresh' => true,
])
);
}
public function build(string $index, array $properties): void
{
$this->client->indices()->create([
'index' => $index,
'body' => [
'settings' => [
'index.max_ngram_diff' => 10,
'analysis' => [
'analyzer' => [
'flarum_analyzer' => [
'type' => $this->settings->get('blomstra-search.analyzer-language') ?: 'english',
],
'flarum_analyzer_partial' => [
'type' => 'custom',
'tokenizer' => 'standard',
'filter' => [
'lowercase',
'partial_search_filter',
],
],
],
'filter' => [
'partial_search_filter' => [
'type' => 'ngram',
'min_gram' => 1,
'max_gram' => 10,
'token_chars' => ['letter', 'digit', 'symbol'],
],
],
],
],
'mappings' => [
'properties' => $properties,
],
],
]);
}
public function flush(string $index): void
{
$this->client->indices()->delete([
'index' => $index,
'ignore_unavailable' => true,
]);
}
public function handleResponse(array $response): void
{
if (Arr::get($response, 'errors') !== true) {
return;
}
$items = Arr::get($response, 'items');
$error = Arr::get(Arr::first($items), 'index.error.reason');
throw new IndexingException(
"Failed to seed: $error",
$items
);
}
}

View File

@ -1,23 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Search;
use Flarum\Search\AbstractDriver;
class ElasticSearchDriver extends AbstractDriver
{
public static function name(): string
{
return 'blomstra-elasticsearch';
}
}

View File

@ -1,43 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Search;
use Blomstra\Search\Elasticsearch\Builder;
use Closure;
use Flarum\Search\SearchState;
class ElasticSearchState extends SearchState
{
protected Builder $builder;
protected ?Closure $retrieveDatabaseRecordsUsing = null;
public function setBuilder(Builder $builder): void
{
$this->builder = $builder;
}
public function getBuilder(): Builder
{
return $this->builder;
}
public function retrieveDatabaseRecordsUsing(Closure $retrieveDatabaseRecordsUsing): void
{
$this->retrieveDatabaseRecordsUsing = $retrieveDatabaseRecordsUsing;
}
public function getRetrieveDatabaseRecordsUsing(): ?Closure
{
return $this->retrieveDatabaseRecordsUsing;
}
}

View File

@ -1,119 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Search;
use Blomstra\Search\Elasticsearch\Builder;
use Elasticsearch\Client;
use Flarum\Search\Filter\FilterManager;
use Flarum\Search\SearchCriteria;
use Flarum\Search\SearcherInterface;
use Flarum\Search\SearchResults;
use Illuminate\Support\Arr;
use Illuminate\Support\Collection;
use Illuminate\Support\Str;
use Spatie\ElasticsearchQueryBuilder\Sorts\Sort;
abstract class Searcher implements SearcherInterface
{
public function __construct(
protected FilterManager $filters,
/** @var array<callable> */
protected array $mutators,
protected Client $elastic
) {
}
abstract public function index(): string;
public function search(SearchCriteria $criteria): SearchResults
{
$builder = (new Builder($this->elastic))
->index($this->index())
->size($criteria->limit + 1)
->from($criteria->offset);
$state = new ElasticSearchState($criteria->actor, $criteria->isFulltext());
$state->setBuilder($builder);
// Default logic for retrieving database records.
// This is normally overriden by the fulltext filter.
$state->retrieveDatabaseRecordsUsing(function (array $response, SearchCriteria $criteria): Collection {
$results = (new Collection(Arr::get($response, 'hits.hits')))->map(function ($hit) {
$type = $hit['_source']['type'];
$id = Str::after($hit['_source']['id'], "$type:");
return [
'id' => $id,
'score' => Arr::get($hit, '_score'),
'weight' => Arr::get($hit, 'sort.0'),
];
})->sortByDesc('weight');
$ids = $results
->take($criteria->limit)
->pluck('id')
->all();
return $this->getQuery($criteria->actor)
->whereIn('id', $ids)
->orderByRaw('FIELD(id, '.implode(',', $ids).')')
->get();
});
$this->filters->apply($state, $criteria->filters);
$this->applySort($state, $criteria);
foreach ($this->mutators as $mutator) {
$mutator($state, $criteria);
}
// echo json_encode($builder->getParams(), JSON_PRETTY_PRINT);
// exit;
$response = $builder->search();
// header('Content-Type: application/json');
// echo json_encode($response, JSON_PRETTY_PRINT);
// exit;
$areMoreResults = count($response['hits']['hits']) > $criteria->limit;
$callback = $state->getRetrieveDatabaseRecordsUsing();
if (!$callback) {
throw new \RuntimeException('No callback set to retrieve database records');
}
$records = $callback($response, $criteria);
return new SearchResults($records, $areMoreResults);
}
protected function applySort(ElasticSearchState $state, SearchCriteria $criteria): void
{
$sort = $criteria->sort;
if ($criteria->sortIsDefault && !empty($state->getDefaultSort())) {
$sort = $state->getDefaultSort();
}
if (is_callable($sort)) {
$sort($state->getBuilder());
} else {
foreach (($criteria->sort ?? []) as $field => $direction) {
$state->getBuilder()->addSort(new Sort(Str::snake($field), $direction));
}
}
}
}

View File

@ -0,0 +1,27 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Searchers;
use Sbp\Search\Seeders\CommentSeeder;
class CommentPostSearcher extends Searcher
{
protected string|null $seeder = CommentSeeder::class;
public function enabled(): bool
{
$enabled = $this->setting('blomstra-search.search-post-bodies', true);
return boolval($enabled);
}
}

View File

@ -0,0 +1,32 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Searchers;
use Sbp\Search\Seeders\DiscussionSeeder;
class DiscussionSearcher extends Searcher
{
protected string|null $seeder = DiscussionSeeder::class;
public function enabled(): bool
{
$enabled = $this->setting('blomstra-search.search-discussion-subjects', true);
return boolval($enabled);
}
public function boost(): float
{
return 1.5;
}
}

View File

@ -0,0 +1,48 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Searchers;
use Sbp\Search\Seeders\Seeder;
use Flarum\Settings\SettingsRepositoryInterface;
abstract class Searcher
{
protected string|null $seeder = null;
public function type(): string
{
/** @var Seeder $seeder */
$seeder = $this->seeder;
if (empty($seeder)) {
throw new \InvalidArgumentException('Implement type or add $seeder');
}
return (new $seeder())->type();
}
public function enabled(): bool
{
return true;
}
public function boost(): float
{
return 1;
}
protected function setting(string $key, $default = null)
{
return resolve(SettingsRepositoryInterface::class)->get($key, $default);
}
}

View File

@ -0,0 +1,96 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Seeders;
use Sbp\Search\Save\Document;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Api\Serializer\PostSerializer;
use Flarum\Discussion\Discussion;
use Flarum\Post\CommentPost;
use Flarum\Post\Event as Core;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Model;
class CommentSeeder extends Seeder
{
public function type(): string
{
return resolve(PostSerializer::class)->getType(new CommentPost());
}
public function joinRelation(): string
{
return 'post';
}
public function routing(Model $model): string
{
return (string) $model->discussion_id;
}
public function query(): Builder
{
return CommentPost::query()
->where('type', CommentPost::$type);
}
public static function savingOn(Dispatcher $events, callable $callable)
{
$events->listen([
Core\Posted::class,
Core\Revised::class,
Core\Hidden::class,
Core\Restored::class,
], function ($event) use ($callable) {
$callable($event->post);
});
}
public static function deletingOn(Dispatcher $events, callable $callable)
{
$events->listen([
Core\Deleted::class
], function ($event) use ($callable) {
$callable($event->post);
});
}
/** Cached discussion type string (e.g. "discussions") — resolved once per job. */
private ?string $discussionType = null;
private function discussionType(): string
{
return $this->discussionType ??= resolve(DiscussionSerializer::class)->getType(new Discussion());
}
/**
* Post documents only need content for has_child matching and the join
* field to establish the parent-child relationship. All discussion-level
* fields (groups, tags, comment_count, etc.) live on the discussion
* document and are irrelevant here.
*
* @param CommentPost $model
*/
public function toDocument(Model $model): Document
{
return new Document([
'join_field' => ['name' => $this->joinRelation(), 'parent' => "{$this->discussionType()}:{$model->discussion_id}"],
'discussion_id' => $model->discussion_id,
'id' => $this->type().':'.$model->id,
'rawId' => $model->id,
'content' => $model->content,
'is_hidden' => $model->hidden_at !== null,
]);
}
}

View File

@ -0,0 +1,210 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Seeders;
use Sbp\Search\Save\Document;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Discussion\Discussion;
use Flarum\Discussion\Event as Core;
use Flarum\Extension\ExtensionManager;
use Flarum\Group\Group;
use Flarum\Group\Permission;
use Flarum\Tags\Tag;
use FoF\Byobu\Events as Byobu;
use FoF\DiscussionViews\Events\DiscussionWasViewed;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder;
use Illuminate\Support\Collection;
use Illuminate\Database\Eloquent\Model;
class DiscussionSeeder extends Seeder
{
public function type(): string
{
return resolve(DiscussionSerializer::class)->getType(new Discussion());
}
public function joinRelation(): string
{
return 'discussion';
}
public function routing(Model $model): string
{
return (string) $model->id;
}
public function query(): Builder
{
return Discussion::query()
->whereNull('hidden_at');
}
public function relationships(): array
{
$includes = [];
if ($this->extensionEnabled('flarum-tags')) {
$includes[] = 'tags';
}
if ($this->extensionEnabled('fof-byobu')) {
$includes[] = 'recipientUsers';
$includes[] = 'recipientGroups';
}
return $includes;
}
public static function savingOn(Dispatcher $events, callable $callable)
{
$events->listen([
// flarum/core events
Core\Started::class, Core\Restored::class, Core\Renamed::class,
// fof/byobu discussion recipients events.
Byobu\DiscussionMadePublic::class, Byobu\RemovedSelf::class, Byobu\RecipientsChanged::class,
], function ($event) use ($callable) {
return $callable($event->discussion);
});
}
public static function viewingOn(Dispatcher $events, callable $callable): void
{
if (!resolve(ExtensionManager::class)->isEnabled('fof-discussion-views')) {
return;
}
$events->listen(DiscussionWasViewed::class, function (DiscussionWasViewed $event) use ($callable) {
$viewCount = $event->discussion->view_count;
$shouldSync = match (true) {
$viewCount < 15 => true,
$viewCount < 100 => rand(1, 3) === 1,
default => rand(1, 19) === 1,
};
if ($shouldSync) {
$callable($event->discussion->id);
}
});
}
public static function deletingOn(Dispatcher $events, callable $callable)
{
$events->listen([
// flarum/core events.
Core\Deleted::class, Core\Hidden::class
], function ($event) use ($callable) {
return $callable($event->discussion);
});
}
/**
* @param Discussion $model
*
* @return Document
*/
public function toDocument(Model $model): Document
{
$document = new Document([
'join_field' => $this->joinRelation(),
'id' => $this->type().':'.$model->id,
'rawId' => $model->id,
'content' => $model->title,
'title' => $model->title,
'created_at' => $model->created_at?->toAtomString(),
'updated_at' => $model->last_posted_at?->toAtomString(),
'is_private' => $model->is_private,
'user_id' => $model->user_id,
'groups' => $this->groupsForDiscussion($model),
'comment_count' => $model->comment_count,
]);
if ($this->extensionEnabled('flarum-tags')) {
$document['tags'] = $model->tags->pluck('id')->toArray();
}
if ($this->extensionEnabled('fof-byobu')) {
$document['recipient_users'] = $model->recipientUsers
->whereNull('removed_at')
->pluck('id')
->toArray();
$document['recipient_groups'] = $model->recipientGroups
->whereNull('removed_at')
->pluck('id')
->toArray();
}
if ($this->extensionEnabled('flarum-sticky')) {
$document['is_sticky'] = (bool) $model->is_sticky;
}
if ($this->extensionEnabled('fof-discussion-views')) {
$document['view_count'] = (int) ($model->view_count ?? 0);
}
return $document;
}
/**
* All viewForum permissions keyed by permission string, loaded once per seeder instance.
* Avoids N×2 Permission queries inside the per-document map loop.
*/
private ?Collection $cachedPermissions = null;
private ?Collection $cachedGlobalPermission = null;
private function allPermissions(): Collection
{
if ($this->cachedPermissions === null) {
$this->cachedPermissions = Permission::query()
->where(function ($q) {
$q->where('permission', 'viewForum')
->orWhere('permission', 'like', 'tag%.viewForum');
})
->get();
$this->cachedGlobalPermission = $this->cachedPermissions
->where('permission', 'viewForum')
->pluck('group_id');
}
return $this->cachedPermissions;
}
protected function groupsForDiscussion(Discussion $discussion): array
{
$allPerms = $this->allPermissions();
$permissions = collect();
if ($this->extensionEnabled('flarum-tags')) {
/** @var Collection $tags */
$tags = $discussion->tags;
$permissions = $tags->map(function (Tag $tag) use ($allPerms) {
$tagPerms = $allPerms->where('permission', "tag$tag->id.viewForum");
if ($tag->is_restricted) {
$tagPerms = $tagPerms->add(['group_id' => Group::ADMINISTRATOR_ID]);
}
return $tagPerms->pluck('group_id');
})->flatten();
}
if (!$discussion->is_private && $permissions->isEmpty()) {
$permissions = $this->cachedGlobalPermission;
}
return $permissions->toArray();
}
}

61
src/Seeders/Seeder.php Normal file
View File

@ -0,0 +1,61 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Seeders;
use Sbp\Search\Save\Document;
use Flarum\Extension\ExtensionManager;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Model;
abstract class Seeder
{
abstract public function type(): string;
/**
* The join relation name for this document type ('discussion' or 'post').
* Used in the join_field mapping and for filtering in seed-missing checks.
*/
abstract public function joinRelation(): string;
/**
* The ES routing key for this model. Parent and child documents for the same
* discussion must share a routing key so ES places them on the same shard.
*/
abstract public function routing(Model $model): string;
abstract public function query(): Builder;
abstract public static function savingOn(Dispatcher $events, callable $callable);
abstract public static function deletingOn(Dispatcher $events, callable $callable);
/** No-op default; override in seeders that need to react to view events. */
public static function viewingOn(Dispatcher $events, callable $callable): void {}
abstract public function toDocument(Model $model): Document;
/** Relationships to eager-load on the collection before calling toDocument(). */
public function relationships(): array
{
return [];
}
protected function extensionEnabled(string $extension): bool
{
/** @var ExtensionManager $manager */
$manager = resolve(ExtensionManager::class);
return $manager->isEnabled($extension);
}
}