Phase J: Introduce RankingExpression and apply to repos

safesploit · safesploit · commit 3da5bbee04af · 2026-06-02T23:32:04.000+01:00
Add a new Doogle\Search\RankingExpression utility to centralize SQL ranking snippets (weighted equals/like, non-empty, HTTPS, bounded click boost, capped MySQL full-text, and sum).

Update Site, Image, and Video repositories to use the new expressions, adjust per-vertical weights and quality signals, and replace inline score aggregation.

Add integration tests to verify click-boost capping so clicks cannot dominate relevance, plus unit tests for RankingExpression.

Documentation updated (ARCHITECTURE2.md, README-refactored.md) to describe the deterministic ranking formula and capped boosts.
diff --git a/ARCHITECTURE2.md b/ARCHITECTURE2.md
@@ -1191,8 +1191,56 @@ Tasks:
 - Ranking by Result Type
 - Bounded Click Boost (Bad behaviour: Result with most clicks always wins.)
 - Quality Signals
-- Search Analytics
-- Admin Ranking Settings (an analytics auth admin portal for managing/viewing ranking)
+
+Implemented ranking formula:
+
+```text
+rankingScore =
+  cappedFullTextBoost
+  + fieldRelevance
+  + qualitySignals
+  + boundedClickBoost
+```
+
+Bounded click boost:
+
+```text
+min(clicks, 100) * 0.1
+```
+
+This makes click boost useful as a tie-breaker and weak popularity signal, but
+caps it at 10 points so clicks cannot dominate stronger textual relevance.
+
+MySQL/MariaDB full-text boost is also capped:
+
+```text
+min(fullTextScore * 50, 120)
+```
+
+Deterministic ordering:
+
+```sql
+ORDER BY rankingScore DESC, clicks DESC, id DESC
+```
+
+Ranking by result type:
+
+- Sites prioritise exact/partial title, then keywords, description, URL, HTTPS, and metadata completeness.
+- Images prioritise exact/partial title, then alt text, image URL, HTTPS, and available title/alt text.
+- Videos prioritise exact/partial title, then description, video URL, site URL, HTTPS, thumbnail availability, and metadata completeness.
+
+Future optional work:
+
+- Search analytics beyond click telemetry.
+- Authenticated admin ranking settings for viewing/tuning ranking weights.
+
+Acceptance criteria:
+
+- all search verticals use the same ranking model shape
+- click counts are capped and cannot dominate stronger relevance
+- quality signals are included
+- ranking order is deterministic
+- repository tests cover ranking behaviour
 
 ---
 
diff --git a/README-refactored.md b/README-refactored.md
@@ -23,12 +23,11 @@ Completed from `ARCHITECTURE2.md`:
 - Phase G: Crawl Jobs / History
 - Phase H: Production Hardening
 - Phase I: Videos Search Vertical
+- Phase J: Search Ranking
 
 Remaining:
 
-- Phase J - Search Ranking
-
-<!-- - None currently tracked in `ARCHITECTURE2.md`. -->
+- Phase K: Search analytics and admin ranking controls are future optional work.
 
 Key changes now in place:
 
@@ -37,6 +36,7 @@ Key changes now in place:
 - `app/` contains auth, crawl, database, repository, search, and security classes.
 - Legacy root `crawl.php`, root `crawl-manual.php`, and `classes/` have been removed.
 - Search uses repositories, services, DTOs, full-text indexes, relevance ranking, and bounded click boost for sites, images, and videos.
+- Ranking uses a shared deterministic formula with capped click/full-text boosts and per-vertical quality signals.
 - Browser crawling requires an admin login and CSRF token.
 - CLI crawling does not require a browser session, but still enforces crawler safety policy.
 - Web and CLI crawl jobs are stored in `crawl_jobs` and shown on the authenticated crawl page.
diff --git a/app/Repository/ImageRepository.php b/app/Repository/ImageRepository.php
@@ -4,6 +4,7 @@
 
 namespace Doogle\Repository;
 
+use Doogle\Search\RankingExpression;
 use PDO;
 use PDOStatement;
 
@@ -92,21 +93,30 @@ private function imageWhereSql(): string
     private function imageRankingSql(): string
     {
         $scores = [
-            'CASE WHEN title LIKE :rankTitleTerm THEN 100 ELSE 0 END',
-            'CASE WHEN alt LIKE :rankAltTerm THEN 60 ELSE 0 END',
-            'CASE WHEN imageUrl LIKE :rankImageUrlTerm THEN 20 ELSE 0 END',
-            '(CASE WHEN clicks > 100 THEN 100 ELSE clicks END * 0.1)',
+            RankingExpression::weightedEquals('title', ':rankTitleExactTerm', 220),
+            RankingExpression::weightedEquals('alt', ':rankAltExactTerm', 180),
+            RankingExpression::weightedLike('title', ':rankTitleTerm', 100),
+            RankingExpression::weightedLike('alt', ':rankAltTerm', 75),
+            RankingExpression::weightedLike('imageUrl', ':rankImageUrlTerm', 25),
+            RankingExpression::httpsUrl('imageUrl', 3),
+            RankingExpression::nonEmpty('alt', 5),
+            RankingExpression::nonEmpty('title', 3),
+            RankingExpression::boundedClickBoost(),
         ];
 
         if ($this->isMysql()) {
             array_unshift(
                 $scores,
-                '(MATCH(' . self::IMAGE_FULL_TEXT_COLUMNS . ') '
-                    . 'AGAINST (:rankFullTextTerm IN NATURAL LANGUAGE MODE) * 50)'
+                RankingExpression::boundedMysqlFullTextBoost(
+                    self::IMAGE_FULL_TEXT_COLUMNS,
+                    ':rankFullTextTerm',
+                    50,
+                    120
+                )
             );
         }
 
-        return '(' . implode(' + ', $scores) . ')';
+        return RankingExpression::sum($scores);
     }
 
     private function bindImageWhereTerms(PDOStatement $statement, string $term): void
@@ -128,6 +138,8 @@ private function bindImageRankingTerms(PDOStatement $statement, string $term): v
         }
 
         $likeTerm = '%' . $term . '%';
+        $statement->bindValue(':rankTitleExactTerm', $term);
+        $statement->bindValue(':rankAltExactTerm', $term);
         $statement->bindValue(':rankTitleTerm', $likeTerm);
         $statement->bindValue(':rankAltTerm', $likeTerm);
         $statement->bindValue(':rankImageUrlTerm', $likeTerm);
diff --git a/app/Repository/SiteRepository.php b/app/Repository/SiteRepository.php
@@ -4,6 +4,7 @@
 
 namespace Doogle\Repository;
 
+use Doogle\Search\RankingExpression;
 use PDO;
 use PDOStatement;
 
@@ -83,22 +84,30 @@ private function siteWhereSql(): string
     private function siteRankingSql(): string
     {
         $scores = [
-            'CASE WHEN title LIKE :rankTitleTerm THEN 100 ELSE 0 END',
-            'CASE WHEN keywords LIKE :rankKeywordsTerm THEN 60 ELSE 0 END',
-            'CASE WHEN description LIKE :rankDescriptionTerm THEN 30 ELSE 0 END',
-            'CASE WHEN url LIKE :rankUrlTerm THEN 20 ELSE 0 END',
-            '(CASE WHEN clicks > 100 THEN 100 ELSE clicks END * 0.1)',
+            RankingExpression::weightedEquals('title', ':rankTitleExactTerm', 240),
+            RankingExpression::weightedLike('title', ':rankTitleTerm', 100),
+            RankingExpression::weightedLike('keywords', ':rankKeywordsTerm', 60),
+            RankingExpression::weightedLike('description', ':rankDescriptionTerm', 35),
+            RankingExpression::weightedLike('url', ':rankUrlTerm', 25),
+            RankingExpression::httpsUrl('url', 5),
+            RankingExpression::nonEmpty('title', 5),
+            RankingExpression::nonEmpty('description', 3),
+            RankingExpression::boundedClickBoost(),
         ];
 
         if ($this->isMysql()) {
             array_unshift(
                 $scores,
-                '(MATCH(' . self::SITE_FULL_TEXT_COLUMNS . ') '
-                    . 'AGAINST (:rankFullTextTerm IN NATURAL LANGUAGE MODE) * 50)'
+                RankingExpression::boundedMysqlFullTextBoost(
+                    self::SITE_FULL_TEXT_COLUMNS,
+                    ':rankFullTextTerm',
+                    50,
+                    120
+                )
             );
         }
 
-        return '(' . implode(' + ', $scores) . ')';
+        return RankingExpression::sum($scores);
     }
 
     private function bindSiteWhereTerms(PDOStatement $statement, string $term): void
@@ -121,6 +130,7 @@ private function bindSiteRankingTerms(PDOStatement $statement, string $term): vo
         }
 
         $likeTerm = '%' . $term . '%';
+        $statement->bindValue(':rankTitleExactTerm', $term);
         $statement->bindValue(':rankTitleTerm', $likeTerm);
         $statement->bindValue(':rankKeywordsTerm', $likeTerm);
         $statement->bindValue(':rankDescriptionTerm', $likeTerm);
diff --git a/app/Repository/VideoRepository.php b/app/Repository/VideoRepository.php
@@ -4,6 +4,7 @@
 
 namespace Doogle\Repository;
 
+use Doogle\Search\RankingExpression;
 use PDO;
 use PDOStatement;
 
@@ -83,22 +84,31 @@ private function videoWhereSql(): string
     private function videoRankingSql(): string
     {
         $scores = [
-            'CASE WHEN title LIKE :rankTitleTerm THEN 100 ELSE 0 END',
-            'CASE WHEN description LIKE :rankDescriptionTerm THEN 60 ELSE 0 END',
-            'CASE WHEN videoUrl LIKE :rankVideoUrlTerm THEN 30 ELSE 0 END',
-            'CASE WHEN siteUrl LIKE :rankSiteUrlTerm THEN 10 ELSE 0 END',
-            '(CASE WHEN clicks > 100 THEN 100 ELSE clicks END * 0.1)',
+            RankingExpression::weightedEquals('title', ':rankTitleExactTerm', 230),
+            RankingExpression::weightedLike('title', ':rankTitleTerm', 100),
+            RankingExpression::weightedLike('description', ':rankDescriptionTerm', 55),
+            RankingExpression::weightedLike('videoUrl', ':rankVideoUrlTerm', 25),
+            RankingExpression::weightedLike('siteUrl', ':rankSiteUrlTerm', 15),
+            RankingExpression::httpsUrl('videoUrl', 3),
+            RankingExpression::nonEmpty('thumbnailUrl', 5),
+            RankingExpression::nonEmpty('title', 5),
+            RankingExpression::nonEmpty('description', 3),
+            RankingExpression::boundedClickBoost(),
         ];
 
         if ($this->isMysql()) {
             array_unshift(
                 $scores,
-                '(MATCH(' . self::VIDEO_FULL_TEXT_COLUMNS . ') '
-                    . 'AGAINST (:rankFullTextTerm IN NATURAL LANGUAGE MODE) * 50)'
+                RankingExpression::boundedMysqlFullTextBoost(
+                    self::VIDEO_FULL_TEXT_COLUMNS,
+                    ':rankFullTextTerm',
+                    50,
+                    120
+                )
             );
         }
 
-        return '(' . implode(' + ', $scores) . ')';
+        return RankingExpression::sum($scores);
     }
 
     private function bindVideoWhereTerms(PDOStatement $statement, string $term): void
@@ -121,6 +131,7 @@ private function bindVideoRankingTerms(PDOStatement $statement, string $term): v
         }
 
         $likeTerm = '%' . $term . '%';
+        $statement->bindValue(':rankTitleExactTerm', $term);
         $statement->bindValue(':rankTitleTerm', $likeTerm);
         $statement->bindValue(':rankDescriptionTerm', $likeTerm);
         $statement->bindValue(':rankVideoUrlTerm', $likeTerm);
diff --git a/app/Search/RankingExpression.php b/app/Search/RankingExpression.php
@@ -0,0 +1,54 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Doogle\Search;
+
+final class RankingExpression
+{
+    public const CLICK_BOOST_CAP = 100;
+    public const CLICK_BOOST_WEIGHT = 0.1;
+
+    public static function sum(array $expressions): string
+    {
+        return '(' . implode(' + ', $expressions) . ')';
+    }
+
+    public static function weightedEquals(string $column, string $parameter, int $weight): string
+    {
+        return "CASE WHEN LOWER({$column}) = LOWER({$parameter}) THEN {$weight} ELSE 0 END";
+    }
+
+    public static function weightedLike(string $column, string $parameter, int $weight): string
+    {
+        return "CASE WHEN {$column} LIKE {$parameter} THEN {$weight} ELSE 0 END";
+    }
+
+    public static function nonEmpty(string $column, int $weight): string
+    {
+        return "CASE WHEN {$column} <> '' THEN {$weight} ELSE 0 END";
+    }
+
+    public static function httpsUrl(string $column, int $weight): string
+    {
+        return "CASE WHEN {$column} LIKE 'https://%' THEN {$weight} ELSE 0 END";
+    }
+
+    public static function boundedClickBoost(string $column = 'clicks'): string
+    {
+        return '(CASE WHEN ' . $column . ' > ' . self::CLICK_BOOST_CAP
+            . ' THEN ' . self::CLICK_BOOST_CAP . ' ELSE ' . $column . ' END * '
+            . self::CLICK_BOOST_WEIGHT . ')';
+    }
+
+    public static function boundedMysqlFullTextBoost(
+        string $columns,
+        string $parameter,
+        int $weight,
+        int $cap,
+    ): string {
+        $score = 'MATCH(' . $columns . ') AGAINST (' . $parameter . ' IN NATURAL LANGUAGE MODE) * ' . $weight;
+
+        return 'LEAST((' . $score . '), ' . $cap . ')';
+    }
+}
diff --git a/tests/Integration/Repository/ImageRepositoryTest.php b/tests/Integration/Repository/ImageRepositoryTest.php
@@ -77,6 +77,18 @@ public function testSearchRanksTitleMatchesAboveHigherClickedAltMatches(): void
         self::assertSame('https://example.com/alt.png', $results[1]['imageUrl']);
     }
 
+    public function testSearchCapsClickBoostSoClicksCannotDominateStrongerRelevance(): void
+    {
+        $this->insertImage('https://example.com/site', 'https://example.com/exact.png', '', 'linux', 0, 0);
+        $this->insertImage('https://example.com/site', 'https://example.com/clicked.png', 'linux', 'Clicked', 10000, 0);
+
+        $results = $this->repository->search('linux', 0, 30);
+
+        self::assertSame('https://example.com/exact.png', $results[0]['imageUrl']);
+        self::assertSame('https://example.com/clicked.png', $results[1]['imageUrl']);
+        self::assertGreaterThan((float) $results[1]['rankingScore'], (float) $results[0]['rankingScore']);
+    }
+
     public function testSearchCanMatchImageUrl(): void
     {
         $this->insertImage(
diff --git a/tests/Integration/Repository/SiteRepositoryTest.php b/tests/Integration/Repository/SiteRepositoryTest.php
@@ -75,6 +75,18 @@ public function testSearchRanksTitleMatchesAboveHigherClickedDescriptionMatches(
         self::assertSame('https://example.com/description', $results[1]['url']);
     }
 
+    public function testSearchCapsClickBoostSoClicksCannotDominateStrongerRelevance(): void
+    {
+        $this->insertSite('https://example.com/exact', 'linux', '', '', 0);
+        $this->insertSite('https://example.com/clicked', 'Clicked Result', 'linux', '', 10000);
+
+        $results = $this->repository->search('linux', 0, 20);
+
+        self::assertSame('https://example.com/exact', $results[0]['url']);
+        self::assertSame('https://example.com/clicked', $results[1]['url']);
+        self::assertGreaterThan((float) $results[1]['rankingScore'], (float) $results[0]['rankingScore']);
+    }
+
     public function testSearchAppliesOffsetAndLimit(): void
     {
         $this->insertSite('https://example.com/one', 'Linux One', 'Linux', 'linux', 3);
diff --git a/tests/Integration/Repository/VideoRepositoryTest.php b/tests/Integration/Repository/VideoRepositoryTest.php
@@ -65,6 +65,18 @@ public function testSearchRanksTitleMatchesAboveHigherClickedDescriptionMatches(
         self::assertSame('https://example.com/description.mp4', $results[1]['videoUrl']);
     }
 
+    public function testSearchCapsClickBoostSoClicksCannotDominateStrongerRelevance(): void
+    {
+        $this->insertVideo('https://example.com/exact.mp4', 'linux', '', 0);
+        $this->insertVideo('https://example.com/clicked.mp4', 'Clicked Video', 'linux', 10000);
+
+        $results = $this->repository->search('linux', 0, 24);
+
+        self::assertSame('https://example.com/exact.mp4', $results[0]['videoUrl']);
+        self::assertSame('https://example.com/clicked.mp4', $results[1]['videoUrl']);
+        self::assertGreaterThan((float) $results[1]['rankingScore'], (float) $results[0]['rankingScore']);
+    }
+
     public function testSearchCanMatchVideoUrl(): void
     {
         $this->insertVideo('https://example.com/linux-demo.mp4', 'Demo', 'Video', 1);
diff --git a/tests/Unit/Search/RankingExpressionTest.php b/tests/Unit/Search/RankingExpressionTest.php
