Commit Graph

157 Commits

Author SHA1 Message Date
Bart van Bragt 557604e426 fix: retry fill ES queries on transient connection failure 2026-04-15 14:07:43 +02:00
Bart van Bragt 08353d2473 perf: sort by date by default, skip scoring on field sorts
- Default sort changed from relevance to updated_at desc. This lets
  has_child use score_mode:none, which skips child scoring entirely
  and lets ES short-circuit early on large corpora.
- When a field sort is requested, score_mode is also none for the same
  reason. score_mode:sum is only used when sorting by relevance.
- track_total_hits:false avoids a full-index count on every query,
  allowing ES to stop once it has collected enough results.
- Strip all gambit operators (tag:foo, author:bar, is:unread, etc.)
  from the ES query string, not just is:private. Leaving them in caused
  operator:and to require the gambit tokens to appear literally in post
  content, producing zero results when gambits were combined with text.
2026-04-14 22:40:43 +02:00
Bart van Bragt 80357e6a77 fix: include --staging in build conflict suggestions 2026-04-14 16:46:06 +02:00
Bart van Bragt d8eebb183a fix: prevent ES CPU saturation from has_child scoring all posts
Without minimum_should_match=1 on the inner post bool query, Elasticsearch
defaults MSM to 0 whenever a filter clause is present. This caused has_child
to score every non-hidden post on every search request, saturating CPU on
shared ES nodes with large corpora (3.7M+ posts).

- Add BoolQuery subclass with create() override and minimumShouldMatch()
  (spatie/elasticsearch-query-builder 1.x uses `new self()` in create(),
  so subclassing requires overriding it)
- Set minimumShouldMatch(1) on the inner post query after adding the
  is_hidden filter, so only genuinely matching posts are scored
- Remove the operator('or') clause from buildShouldClauses() — with
  Turkish min_ngram=2, 'or' generates 2-gram tokens that match nearly
  every post, causing near-total index scans
- Add ES client timeouts (connect: 2s, query: 10s) to prevent Apache
  mod_php worker saturation when ES is slow or unreachable
2026-04-14 16:07:50 +02:00
Bart van Bragt c54f37349f fix: use Support\Collection for cachedGlobalPermission — pluck() returns plain Collection, not Eloquent 2026-04-14 11:27:02 +02:00
Bart van Bragt ca8c248870 fix: use Eloquent Collection in job — loadMissing does not exist on Support\Collection 2026-04-14 09:59:02 +02:00
Bart van Bragt bf37e01eb0 fix: read forum attribute lazily inside extend callback 2026-04-14 09:33:17 +02:00
Bart van Bragt 4709b9d1db fix: is_hidden boolean coercion and forum settings access 2026-04-14 08:59:19 +02:00
Bart van Bragt 2fc66b9bf7 perf: eager-load relationships in UpdateSearchJob and cache permissions in DiscussionSeeder
- Add relationships() to Seeder base class; DiscussionSeeder implements it to
  return tags/recipientUsers/recipientGroups for eager loading
- UpdateSearchJob calls loadMissing(relationships()) before the map loop,
  eliminating N×R lazy-load queries per batch
- Remove refresh:true from bulk call — synchronous ES refresh was the primary
  throughput bottleneck during indexing
- Cache viewForum permissions in DiscussionSeeder per job instance, replacing
  N×2 Permission queries per document with a single cached query
- Increase seeder batch size from 1000 to 2500
2026-04-13 22:18:14 +02:00
Bart van Bragt 439c74f0a3 fix: store index compat version in ES mapping _meta for correct rollback behaviour
- Write INDEX_COMPAT_VERSION into mapping _meta.index_compat_version on every
  build/promote so the version travels with the index rather than being a
  separate settings value
- saveIndexedConfig now reads compat version back from _meta alongside the
  existing analyzer/min_gram sync, so rollback automatically restores the
  correct value (or null for pre-feature indexes, triggering the reindex warning)
- Replace InvalidArgumentException in buildIndexSettings with $this->error()
  fallback to avoid stack traces in the console
- Fix stale comment in ViewsSearchJob referencing removed CLI flags
2026-04-10 13:46:08 +02:00
Bart van Bragt f486292f15 feat: make ngram min_gram configurable via admin setting
Adds a 'Minimum search query length' setting (1–4, default 3). The admin
shows a warning when the analyzer or min-search-length have changed since
the last index build.
2026-04-10 13:19:22 +02:00
Bart van Bragt 4412583cd9 chore: fix prettier formatting and npm audit vulnerabilities
Reformat admin/index.ts to satisfy prettier checks. Update package-lock.json
via npm audit fix to resolve 17 vulnerabilities in transitive dev dependencies
(webpack, babel, lodash, ws, semver, etc.).
2026-04-10 11:47:38 +02:00
Bart van Bragt 81e7b6502d feat: Show admin dashboard notice when search index needs rebuilding
After a mapping change (e.g. the parent-child restructure), existing installs
need to rebuild their index. A DashboardWidget appears on the admin page
warning the admin and showing the command to run. Compatibility is tracked
via the blomstra-search.index-compatible setting.
2026-04-10 11:10:53 +02:00
Bart van Bragt 3039b65946 refactor: Simplify index command with better naming/options
The flag-based interface (--rebuild etc) became rather convoluted with the changes. Replace this with commands with more straight forward naming.

Also added an explicit 'rollback' action and updated the README.md
2026-04-10 09:35:49 +02:00
Bart van Bragt e2e2d1428f feat: restructure index as parent-child documents with blue-green rebuilds
- Switch to ES parent/child join_field: discussion docs hold all metadata,
  post docs hold content only. Routing ensures parent and child land on the
  same shard. Removes the old flat-document approach where post data was
  duplicated onto every comment document.

- Add HasChildQuery with inner_hits so the best-matching post ID is
  surfaced as mostRelevantPost without a second DB query.

- Blue-green index rebuilds: --recreate writes into a timestamped pending
  index, --swap atomically promotes it via alias. Interrupted builds are
  resumable with --recreate --continue. --swap requires confirmation before
  proceeding.

- Index hidden posts; non-moderators are filtered at query time via
  is_hidden on the has_child clause. Core\Hidden and Core\Restored trigger
  re-indexing so moderators with post.hide can search hidden posts.
2026-04-09 23:30:57 +02:00
Bart van Bragt 80d66c98c4 feat: index and sync view_count for fof/discussion-views sort support
- Rename SavingJob to UpdateSearchJob
- Add ViewsSearchJob for lightweight partial view_count updates
- Add probabilistic sync throttle in DiscussionSeeder::viewingOn()
  to avoid an ES update on every page view
- Add view_count to mapping, DiscussionSeeder::toDocument(), and
  translateSort
- Log a warning and skip instead of 500ing on unknown sort fields
2026-04-09 12:05:28 +02:00
Bart van Bragt ff2a252e99 fix: editing a post does not update the search index
CommentSeeder only listened to Post\Event\Posted, so post edits
left the Elasticsearch document stale.
2026-04-09 11:17:17 +02:00
Bart van Bragt 5929a191f6 fix: --seed-missing stops early when a full range is already indexed
When all documents in a range were already in the index, the empty
collection caused continueAt to be set to null, aborting before
scanning lower ID ranges. Now advances past the range bottom instead.
2026-04-09 11:17:17 +02:00
Bart van Bragt 6699c8cb58 fix: ngram min_gram=1 indexes tokens that can never be matched
Flarum's search UI enforces a minimum query length of 3 characters
(MIN_SEARCH_LEN in Search.tsx), so 1- and 2-character ngram tokens
are indexed but can never be queried. Changed min_gram from 1 to 3
and updated max_ngram_diff accordingly.
2026-04-09 11:04:27 +02:00
Bart van Bragt 369ee504f3 fix: remove unused content_partial field from seeders
content_partial was introduced alongside a partialMatch() query that
targeted it. That query was later switched to use the content field
instead, but content_partial was never removed from the seeders,
causing every document to store and index its content twice.
2026-04-09 08:39:58 +02:00
Bart van Bragt eeb091273d fix: tag filter produces empty results when browsing a tag
When navigating to /t/{tag}?q=query the tag filter from the request
was silently ignored. Added a TermsQuery on the tags field using the
tag IDs resolved from the provided slugs.
2026-04-09 08:20:42 +02:00
Daniël Klabbers d252a17586 fix: not changing recipients visibility when recipients change 2024-06-13 19:27:14 +02:00
Daniël Klabbers a9dc8bfb16 fix: byobu discussions open up while using this extension 2024-06-12 11:34:18 +02:00
StyleCI Bot 82e6ec786f
Apply fixes from StyleCI 2023-10-18 13:57:41 +00:00
flarum-bot f981c27e00 Bundled output for commit f6503f5e6a
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2023-01-30 22:17:40 +00:00
Daniël Klabbers f6503f5e6a fix: #10, private forums have no discussions key, throws error 2023-01-30 23:17:01 +01:00
flarum-bot ad92d8856b Bundled output for commit 8b519619e5
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-24 09:39:12 +00:00
StyleCI Bot 8b519619e5
Apply fixes from StyleCI 2022-11-24 09:38:21 +00:00
Daniël Klabbers 83afe95d61 chore: fix search and seeding 2022-11-24 10:37:57 +01:00
Daniël Klabbers 846cc348ff Merge branch 'dk/search-improvements' into main 2022-11-24 10:29:18 +01:00
Daniël Klabbers 508bb42ac9 chore: clean up buildcommand 2022-11-23 12:46:04 +01:00
StyleCI Bot db74da3e37
Apply fixes from StyleCI 2022-11-23 11:29:48 +00:00
Daniël Klabbers a02d35394b feat: store tags, rawId, support indexing missing objects 2022-11-23 12:09:10 +01:00
flarum-bot b60563f2ec Bundled output for commit 703cbbdd48
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-21 15:33:18 +00:00
David Wheatley 703cbbdd48
fix: incorrect JS package name 2022-11-21 15:32:22 +00:00
flarum-bot 769b6cd278 Bundled output for commit f500d7f956
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-21 12:38:34 +00:00
Ian Morland f500d7f956 fix: revert back to not using the elastic endpoint for filtering, at least until we support tags in the search database 2022-11-21 12:37:52 +00:00
flarum-bot 7257d5fb34 Bundled output for commit 09fb4e8b7b
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-18 09:39:40 +00:00
Daniël Klabbers 09fb4e8b7b
Merge pull request #9 from blomstra/im/fix-search-override
fix: override search not fired correctly
2022-11-18 10:38:47 +01:00
Ian Morland 21170fbe1b Merge remote-tracking branch 'origin/main' into im/fix-search-override 2022-11-18 09:37:37 +00:00
StyleCI Bot 1cd3dac177
Apply fixes from StyleCI 2022-11-18 09:34:30 +00:00
Ian Morland 5815144d3a Revert "Apply fixes from StyleCI"
This reverts commit a26f88a9b6.
2022-11-18 09:34:16 +00:00
StyleCI Bot f8aa3dcf13
Apply fixes from StyleCI 2022-11-18 09:33:13 +00:00
StyleCI Bot a26f88a9b6
Apply fixes from StyleCI 2022-11-18 09:29:46 +00:00
Ian Morland 23dd403417 fix: override endpoint not used correctly 2022-11-18 09:29:19 +00:00
Ian Morland ed718e1458 chore: fix incorect command 2022-11-18 09:28:52 +00:00
flarum-bot 4240269b7f Bundled output for commit c186535bbb
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-18 08:31:43 +00:00
Ian Morland c186535bbb chore: use npm 2022-11-18 08:30:44 +00:00
Ian Morland fa90def995 chore: update frontend workflow 2022-11-18 08:29:19 +00:00
Ian Morland 4d2c5cb22f chore: bump deps, npm audit fix 2022-11-18 08:26:26 +00:00