Commit Graph

100 Commits

Author SHA1 Message Date
Bart van Bragt c5643bed31 fix: default search sort to latest and keep dropdown label in sync
- Search with no explicit sort now defaults to latest (updated_at desc)
  on both backend and frontend, matching the forum UX expectation.
- "All discussions" link from the search box routes to /?q=...&sort=latest
  so the sort dropdown shows "Latest" immediately after submission.
- extendDiscussionState injects sort=-lastPostedAt into API params when
  no sort is present, keeping direct-URL navigation (?q=anan) consistent
  with the backend default.
- Backend guard changed from phpSortField===null to empty($sorts), which
  is the correct sentinel since $sorts is the authoritative ES sort list.
2026-04-15 17:21:10 +02:00
Bart van Bragt 6bcd2becd5 fix: report actual failed item error in bulk seeding exception 2026-04-15 14:08:04 +02:00
Bart van Bragt 2706ed6be0 fix: index discussion on rename 2026-04-15 14:07:55 +02:00
Bart van Bragt 557604e426 fix: retry fill ES queries on transient connection failure 2026-04-15 14:07:43 +02:00
Bart van Bragt 08353d2473 perf: sort by date by default, skip scoring on field sorts
- Default sort changed from relevance to updated_at desc. This lets
  has_child use score_mode:none, which skips child scoring entirely
  and lets ES short-circuit early on large corpora.
- When a field sort is requested, score_mode is also none for the same
  reason. score_mode:sum is only used when sorting by relevance.
- track_total_hits:false avoids a full-index count on every query,
  allowing ES to stop once it has collected enough results.
- Strip all gambit operators (tag:foo, author:bar, is:unread, etc.)
  from the ES query string, not just is:private. Leaving them in caused
  operator:and to require the gambit tokens to appear literally in post
  content, producing zero results when gambits were combined with text.
2026-04-14 22:40:43 +02:00
Bart van Bragt 80357e6a77 fix: include --staging in build conflict suggestions 2026-04-14 16:46:06 +02:00
Bart van Bragt d8eebb183a fix: prevent ES CPU saturation from has_child scoring all posts
Without minimum_should_match=1 on the inner post bool query, Elasticsearch
defaults MSM to 0 whenever a filter clause is present. This caused has_child
to score every non-hidden post on every search request, saturating CPU on
shared ES nodes with large corpora (3.7M+ posts).

- Add BoolQuery subclass with create() override and minimumShouldMatch()
  (spatie/elasticsearch-query-builder 1.x uses `new self()` in create(),
  so subclassing requires overriding it)
- Set minimumShouldMatch(1) on the inner post query after adding the
  is_hidden filter, so only genuinely matching posts are scored
- Remove the operator('or') clause from buildShouldClauses() — with
  Turkish min_ngram=2, 'or' generates 2-gram tokens that match nearly
  every post, causing near-total index scans
- Add ES client timeouts (connect: 2s, query: 10s) to prevent Apache
  mod_php worker saturation when ES is slow or unreachable
2026-04-14 16:07:50 +02:00
Bart van Bragt c54f37349f fix: use Support\Collection for cachedGlobalPermission — pluck() returns plain Collection, not Eloquent 2026-04-14 11:27:02 +02:00
Bart van Bragt ca8c248870 fix: use Eloquent Collection in job — loadMissing does not exist on Support\Collection 2026-04-14 09:59:02 +02:00
Bart van Bragt 4709b9d1db fix: is_hidden boolean coercion and forum settings access 2026-04-14 08:59:19 +02:00
Bart van Bragt 2fc66b9bf7 perf: eager-load relationships in UpdateSearchJob and cache permissions in DiscussionSeeder
- Add relationships() to Seeder base class; DiscussionSeeder implements it to
  return tags/recipientUsers/recipientGroups for eager loading
- UpdateSearchJob calls loadMissing(relationships()) before the map loop,
  eliminating N×R lazy-load queries per batch
- Remove refresh:true from bulk call — synchronous ES refresh was the primary
  throughput bottleneck during indexing
- Cache viewForum permissions in DiscussionSeeder per job instance, replacing
  N×2 Permission queries per document with a single cached query
- Increase seeder batch size from 1000 to 2500
2026-04-13 22:18:14 +02:00
Bart van Bragt 439c74f0a3 fix: store index compat version in ES mapping _meta for correct rollback behaviour
- Write INDEX_COMPAT_VERSION into mapping _meta.index_compat_version on every
  build/promote so the version travels with the index rather than being a
  separate settings value
- saveIndexedConfig now reads compat version back from _meta alongside the
  existing analyzer/min_gram sync, so rollback automatically restores the
  correct value (or null for pre-feature indexes, triggering the reindex warning)
- Replace InvalidArgumentException in buildIndexSettings with $this->error()
  fallback to avoid stack traces in the console
- Fix stale comment in ViewsSearchJob referencing removed CLI flags
2026-04-10 13:46:08 +02:00
Bart van Bragt f486292f15 feat: make ngram min_gram configurable via admin setting
Adds a 'Minimum search query length' setting (1–4, default 3). The admin
shows a warning when the analyzer or min-search-length have changed since
the last index build.
2026-04-10 13:19:22 +02:00
Bart van Bragt 81e7b6502d feat: Show admin dashboard notice when search index needs rebuilding
After a mapping change (e.g. the parent-child restructure), existing installs
need to rebuild their index. A DashboardWidget appears on the admin page
warning the admin and showing the command to run. Compatibility is tracked
via the blomstra-search.index-compatible setting.
2026-04-10 11:10:53 +02:00
Bart van Bragt 3039b65946 refactor: Simplify index command with better naming/options
The flag-based interface (--rebuild etc) became rather convoluted with the changes. Replace this with commands with more straight forward naming.

Also added an explicit 'rollback' action and updated the README.md
2026-04-10 09:35:49 +02:00
Bart van Bragt e2e2d1428f feat: restructure index as parent-child documents with blue-green rebuilds
- Switch to ES parent/child join_field: discussion docs hold all metadata,
  post docs hold content only. Routing ensures parent and child land on the
  same shard. Removes the old flat-document approach where post data was
  duplicated onto every comment document.

- Add HasChildQuery with inner_hits so the best-matching post ID is
  surfaced as mostRelevantPost without a second DB query.

- Blue-green index rebuilds: --recreate writes into a timestamped pending
  index, --swap atomically promotes it via alias. Interrupted builds are
  resumable with --recreate --continue. --swap requires confirmation before
  proceeding.

- Index hidden posts; non-moderators are filtered at query time via
  is_hidden on the has_child clause. Core\Hidden and Core\Restored trigger
  re-indexing so moderators with post.hide can search hidden posts.
2026-04-09 23:30:57 +02:00
Bart van Bragt 80d66c98c4 feat: index and sync view_count for fof/discussion-views sort support
- Rename SavingJob to UpdateSearchJob
- Add ViewsSearchJob for lightweight partial view_count updates
- Add probabilistic sync throttle in DiscussionSeeder::viewingOn()
  to avoid an ES update on every page view
- Add view_count to mapping, DiscussionSeeder::toDocument(), and
  translateSort
- Log a warning and skip instead of 500ing on unknown sort fields
2026-04-09 12:05:28 +02:00
Bart van Bragt ff2a252e99 fix: editing a post does not update the search index
CommentSeeder only listened to Post\Event\Posted, so post edits
left the Elasticsearch document stale.
2026-04-09 11:17:17 +02:00
Bart van Bragt 5929a191f6 fix: --seed-missing stops early when a full range is already indexed
When all documents in a range were already in the index, the empty
collection caused continueAt to be set to null, aborting before
scanning lower ID ranges. Now advances past the range bottom instead.
2026-04-09 11:17:17 +02:00
Bart van Bragt 6699c8cb58 fix: ngram min_gram=1 indexes tokens that can never be matched
Flarum's search UI enforces a minimum query length of 3 characters
(MIN_SEARCH_LEN in Search.tsx), so 1- and 2-character ngram tokens
are indexed but can never be queried. Changed min_gram from 1 to 3
and updated max_ngram_diff accordingly.
2026-04-09 11:04:27 +02:00
Bart van Bragt 369ee504f3 fix: remove unused content_partial field from seeders
content_partial was introduced alongside a partialMatch() query that
targeted it. That query was later switched to use the content field
instead, but content_partial was never removed from the seeders,
causing every document to store and index its content twice.
2026-04-09 08:39:58 +02:00
Bart van Bragt eeb091273d fix: tag filter produces empty results when browsing a tag
When navigating to /t/{tag}?q=query the tag filter from the request
was silently ignored. Added a TermsQuery on the tags field using the
tag IDs resolved from the provided slugs.
2026-04-09 08:20:42 +02:00
Daniël Klabbers d252a17586 fix: not changing recipients visibility when recipients change 2024-06-13 19:27:14 +02:00
Daniël Klabbers a9dc8bfb16 fix: byobu discussions open up while using this extension 2024-06-12 11:34:18 +02:00
StyleCI Bot 8b519619e5
Apply fixes from StyleCI 2022-11-24 09:38:21 +00:00
Daniël Klabbers 83afe95d61 chore: fix search and seeding 2022-11-24 10:37:57 +01:00
Daniël Klabbers 846cc348ff Merge branch 'dk/search-improvements' into main 2022-11-24 10:29:18 +01:00
Daniël Klabbers 508bb42ac9 chore: clean up buildcommand 2022-11-23 12:46:04 +01:00
StyleCI Bot db74da3e37
Apply fixes from StyleCI 2022-11-23 11:29:48 +00:00
Daniël Klabbers a02d35394b feat: store tags, rawId, support indexing missing objects 2022-11-23 12:09:10 +01:00
StyleCI Bot f8aa3dcf13
Apply fixes from StyleCI 2022-11-18 09:33:13 +00:00
Daniël Klabbers 9401b677a3 change weighing for exact words match 2022-11-10 12:57:46 +01:00
Daniël Klabbers 4bc89949ec complete testing 2022-11-10 12:54:26 +01:00
Daniël Klabbers 29ba864849 settings configurations 2022-11-09 18:06:48 +01:00
Daniël Klabbers e26d90d7b6 fix ambiguous 2022-03-04 12:37:43 +01:00
Daniël Klabbers cb3d4f5225 fix typo 2022-02-28 20:33:01 +01:00
Daniël Klabbers c6fd942bf1 prevent leaking deleted discussions 2022-02-28 20:32:27 +01:00
Daniël Klabbers cc8145c236 drop partial search 2022-02-21 10:20:46 +01:00
Daniel Klabbers ccfb964aff improve search results 2022-02-14 16:51:55 +01:00
Daniël Klabbers 133ded0684 fixes #5 2022-01-11 10:10:39 +01:00
Daniel Klabbers 803d1ecfc5 fix partial matching and weighting 2022-01-10 14:16:42 +01:00
Daniel Klabbers 502f7a9f86 use partial match with weighting 2022-01-10 13:47:05 +01:00
Daniel Klabbers 9108553f09 add and vs or distinction, but partial isnt working in combination with the others yet 2022-01-06 12:58:50 +01:00
Daniel Klabbers b6ac018a56 fix partial matching 2022-01-03 16:53:45 +01:00
Daniel Klabbers 88b3e57d8f added weighting properly, also different for titles vs posts 2022-01-03 15:13:13 +01:00
Daniel Klabbers f180d66815 Revert "feat: add partial word match with low weighting"
This reverts commit 0d51e5d912.
2021-12-23 13:07:04 +01:00
Daniel Klabbers 013f5162c5 Revert "introduce wildcard, needs further testing"
This reverts commit 877533b43d.
2021-12-23 13:07:03 +01:00
Daniel Klabbers 23074a3a0c Revert "attempt to make partial match filter"
This reverts commit f21dc04a00.
2021-12-23 13:07:00 +01:00
Daniel Klabbers 0ee902b7b7 Revert "allow minimum should match config"
This reverts commit d5eebf6c65.
2021-12-23 13:06:56 +01:00
Daniel Klabbers d5eebf6c65 allow minimum should match config 2021-12-16 15:05:22 +01:00