Skip to content

v0.9.8 — Phase 1-4 + Hardening v3/v4 + FP-Reduction Sprint

Latest
Latest

Choose a tag to compare

View all tags
@nrodear nrodear released this 09 Jun 22:39
v0.9.8
c99145a

Release 0.9.8 — Phase 1-4 + Hardening v3/v4 + FP-Reduction Sprint

Released: 2026-06-10
🇩🇪 Deutsche Version

This release closes the 0.9.8 development cycle that began on
2026-06-02. Three layers of work landed on top of v0.9.7:

  1. Scanner-Qualität Phases 1-4 (since v0.9.7) — 13 quick-wins,
    confidence audit, A.5 {$IFDEF}-Awareness, SCA165 UnusedSuppression,
    Golden-Corpus regression suite, SARIF partialFingerprints for
    baseline diffs.
  2. Hardening v3 + v4 (2026-06-07/08) — stack-overflow protection
    for deep ASTs (8 iterative-DFS detector refactors + 32 MB stack
    PE-patch), AST-Destroy reentrancy double-free fix, DFM
    resource-wrapper ($FF $0A $00) support, IDE-plugin Win32-OOM fix
    via FixHint memoize-cache, scan.log phase-tracking + skip-log.
  3. FP-Reduction Sprint (2026-06-09) — self-scan FPs in four style
    detectors reduced by 78% (67 → 15). Side-fix: FreeAndNil(Self.Field)
    with Self.-qualifier is now recognised as freeing.

Full real-world verification: 24 repos in
D:\git-sca-realworld\ scan in 411 s with 0 crashes and produce
1.055.283 findings at profile strict.

TL;DR Highlights (delta over the original Phase 1-only plan)

  • DFM Resource-Wrapper-Format $FF $0A $00 now parses correctly —
    GExperts jumps from 0 to 1.084 DFM-findings, JVCL DFM-coverage roughly
    doubles. The previous code mis-classified these as text and silently
    emitted nothing.
  • No more EInvalidPointer/SCA006 mid-scan crash on second file
    via the AST-Destroy reentrancy bug fix (5ecd498).
  • No more Win32 EOutOfMemory in the IDE plugin's
    HighlightAllFindingsInFile at ≥100k findings — TFixHintResolver
    now caches by (Kind, Severity).
  • scan.log Phase-Tracking — every Analyseabbruch: finding now
    reveals the last successful phase + current file in the log,
    replacing the old black box.
  • scan.log skip-log — ignored / excluded files appear with a reason
    instead of disappearing silently.
  • 4 style-detector FP-classes killed without weakening real-bug
    detection: SCA017 DebugOutput (string-literal contents), SCA070 CommentedOutCode (inline-doc + doc-block start), SCA019 TodoComment
    (detector-source mentions), SCA005 FormatMismatch (empty const
    resolve).
  • FieldLeak-Detector — FreeAndNil(Self.Field) with Self.-
    qualifier is now recognised (USepa.pas repro).
  • Two new configuration knobs —
    [Detectors] MaxLineLength=N (default 120) and
    [Detectors] MaxCaseBranches=N (default 10).
  • Anfänger-Build-Guide — HowTo_Build{,_de}.md walks Delphi
    newcomers from IDE installation to first IDE-plugin scan.

Earlier in the 0.9.8 cycle (the Phase 1 deliverables — still in this release)

13 commits since v0.9.7. Phase 1 of
Konzept_ScannerQualitaet.md is
complete (6/6 quick-wins); Phase 4 has begun with the A.3-Minimal
cross-unit visibility check. A multi-persona review (Architecture +
Security + Performance) hardened the code along the way.

Highlights

  • --time-detectors Markdown report — per-detector cumulative
    wall-time + call count for data-driven optimisation.
  • Test-fixture auto-detection — findings from uTest*.pas /
    *Sample.pas / *Demo.pas and test/samples/demos/resources
    directories are filtered out in the default and selftest-quiet
    profiles. Self-test report is now clean of intentional-fixture noise.
  • Unused-suppression tracking (SCA165) — // noinspection X markers
    that never suppressed a finding are themselves flagged. Suppression
    hygiene now visible.
  • Golden-corpus FP-regression suite — 5 historical FP reproducers
    per Round-1..13 fix, PowerShell runner, CI-ready exit code. Every
    future detector change is regression-guarded.
  • SARIF + Baseline contextHash — SHA256 over the ±3-line snippet
    (whitespace-normalised) added to every finding. Baseline matches via
    contextHash or legacy fingerprint — survives method renames and
    small refactors. Old baselines remain valid (no migration needed).
  • Confidence audit (35 kinds → fcMedium) — heuristic / metric /
    style / DFM-schema / no-data-flow-security kinds tagged. With
    --min-confidence high the report shows only structural bugs without
    losing detector coverage.
  • A.3-Minimal: SCA052 cross-unit reactivated — gSymbolRefIndex is
    now consulted for fkUnusedPublicMember. Methods called via
    obj.Method(args) from another unit are no longer flagged as "dead
    public API".
  • Security hardening — noinspection All excludes critical kinds,
    marker parsing is string-context-aware, test-fixture filter is
    repo-root-anchored, baseline JSON has entry + length caps.
  • Performance — gFileTextCache is mtime-aware and lives across
    the post-scan phase; saves ~191k redundant file reads + UTF-8
    validations on a real-world scan. uVisibilityCheck tree-walks cached
    once per unit.

Phase 1 quick-wins (6/6)

C.4 Per-detector performance profile

--time-detectors writes a Markdown table with per-detector
cumulative wall-time, call count, average. Lets you pick optimisation
targets based on data, not gut feeling.

analyser.exe --path D:\repo --full --time-detectors > perf.md

A.2 Test-fixture auto-detection

TDetectorUtils.IsTestFixturePath returns true for paths matching:

  • basenames: uTest*.pas, *_Test.pas, *_Tests.pas, *Sample.pas,
    *Demo.pas, *Sample_*.pas, *_Demo_*.pas, *Sample.dfm,
    *Demo.dfm, MeineUnit.pas (and a few SCA-internal demo files)
  • repo-root-relative directory components:
    test, tests, unittest, unittests, samples, demos,
    resources

Auto-on for --profile default and selftest-quiet, off for strict.
Manual override via --hide-test-fixtures / --show-test-fixtures.

C.3 Unused-suppression tracking (SCA165)

For every // noinspection X marker we record Consumed = false
initially. When ApplyToFindings suppresses a finding via the marker,
Consumed := true. After the suppression pass, any marker still at
Consumed = false emits an fkUnusedSuppression finding on the marker
line, with a quick-fix hint to remove the marker.

C.1 Golden-corpus regression tests

tests/golden-corpus/fp-reproducers/ holds 5 historical false-positive
reproducer .pas snippets (one per Round-1..13 fix). expected.json
maps each file to a must_not_flag list of rule IDs. The PowerShell
runner tools/check-golden-corpus.ps1 scans the corpus and exits 0/1
based on whether any previously-fixed FP class fires again.

powershell -ExecutionPolicy Bypass -File tools\check-golden-corpus.ps1

C.2 SARIF partialFingerprints.contextHash/v1

New unit uFindingFingerprint:

contextHash/v1  =  "v1:" + SHA256( Normalize( ±3 lines around finding ) )

Normalize collapses whitespace runs to a single space, tabs → space,
empty lines dropped, CRLF / LF normalised. The hash is stable against
re-indent, trailing-whitespace cleanup and small line-drift refactors.

uBaseline.Apply matches contextHash or legacy fingerprint —
old baselines without contextHash remain valid (backward-compat),
new baselines survive small refactors.

A.1 Confidence-tagging audit

KindDefaultConfidence in uSCAConsts tags ~35 detector kinds as
fcMedium (pattern-match, metrics, style, DFM heuristics, security
without data-flow). All structural-bug detectors stay fcHigh.
TLeakFinding.SetKind reads from this map. Override pattern (e.g.
uCommandInjection := fcLow) is preserved via the new
SetKind(K, AConfidence) overload. Per-kind justifications in
docs/ConfidenceAudit.md.

Phase 4 partial — A.3 cross-unit visibility (begun)

Minimal step

uVisibilityCheck for fkUnusedPublicMember (SCA052) now consults
gSymbolRefIndex.HasExternalRefs before emitting. If any external unit
references the member via Obj.Member(args), no finding is emitted.

Audit (spot-check 18 known cross-unit methods)

  • ✅ 8/18 (44 %) A.3 takes effect — pattern: instance.Method(args)
  • ❌ 10/18 (56 %) A.3 inactive — 3 known index limitations:
    • nkRef / value-use without parens (F.SeverityText)
    • class-function-call TClass.ClassMethod(args) — AST does not
      classify as Obj.Member
    • bare calls (Fingerprint(F)) — deliberate index simplification

All three are documented in Konzept_ScannerQualitaet.md §A.3+ as the
prioritised follow-up sprint.

Library FP class (not addressable by the index)

Real-world scan shows the top SCA052 hot-spots are vendor libraries
(mormot.*, MVCFramework, LoggerPro) where the consumers live in
other repos that the index cannot see. Recommendation: path-override
default for vendor/ directories.

Multi-persona review

After Phase 1 + A.3-Minimal a three-persona review (Architecture +
Security + Performance) was run on the branch. 9 of 10 findings were
implemented in commit 120894a:

# Area Fix
1 Perf Baseline.Apply skips ContextHash for legacy baselines
2 Sec noinspection All excludes security-critical kinds
3 Sec ParseMarkerLine uses ScanCodeLine (string-context-aware)
4 Sec IsTestFixturePath repo-root-anchored via BaseDir
5 Perf gFileTextCache lives through post-scan + leak fix
6 Sec Baseline JSON 1 M entry / 256 char cap
8 Perf Suppression uses AcquireLines (cache hits)
9 Perf uVisibilityCheck tree-walks cached per unit
10 API SetKind(K, AConfidence) overload

Finding #7 (DefaultConfidence into TFindingKindMeta record) deferred
as invasive (touches ~160 KIND_META entries) and would require an
explicit drift-test design.

TFileTextCache was subsequently made mtime-aware (commit 1e7e193) to
plug a stale-cache issue exposed by the regression suite after the
post-scan cache change.

Files added

  • StaticCodeAnalyserForm/sources/Infrastructure/uFindingFingerprint.pas
  • StaticCodeAnalyserForm/tests/uTestFindingFingerprint.pas
  • tests/golden-corpus/README.md
  • tests/golden-corpus/fp-reproducers/fp01..fp05_*.pas
  • tests/golden-corpus/fp-reproducers/expected.json
  • tools/check-golden-corpus.ps1
  • docs/ConfidenceAudit.md
  • Konzept_ScannerQualitaet.md
  • docs/releases/v0.9.8.md + v0.9.8_de.md (this file)

Files changed

uBaseline, uExportSARIF, uSuppression, uVisibilityCheck,
uDetectorUtils, uMethodd12, uSCAConsts, uConfidenceFilter
(tests), uConsoleRunner, uStaticAnalyzer2, uCommandInjection,
uCustomRuleDetector, uDivByZero, uLeakDetector2,
uFileTextCache, uFindingFilter, uMainForm, uIDEAnalyserForm.

Migration

No breaking changes for callers of the public CLI / SARIF / baseline.

  • Existing baselines work as-is (matched via legacy fingerprint).
  • New baselines additionally carry contextHash. Old Baseline.Apply
    versions ignore the unknown field.
  • Detectors that previously set F.Confidence := xxx after SetKind
    should migrate to SetKind(K, AConfidence) to avoid the order-fragile
    pattern. The old pattern still works.

Commit log (since v0.9.7)

1e7e193  fix(cache):       mtime-aware cache-invalidation in TFileTextCache
2b723f7  fix(build):       IsTestFixturePath impl signature
120894a  fix(review):      9 review findings (Security + Perf + API)
e18323d  refactor:         Clean-code fixes (DRY, SRP, naming)
3054630  fix(visibility):  A.3 OwnUnit path + roadmap update
0ab0bf4  feat(visibility): A.3-Minimal — gSymbolRefIndex for SCA052
a8c7c35  feat(confidence): A.1 audit — ~35 kinds as fcMedium
91ae2ec  feat(baseline):   C.2 SARIF contextHash + baseline match
7b957a8  test(corpus):     C.1 Golden-corpus + runner
c0234d7  feat(suppression):C.3 Unused-suppression tracking (SCA165)
57a0b06  feat(filter):     A.2 Test-fixture auto-detection
1b5a145  fix(perf):        gDetectorTimings in interface section
79b4f56  feat(cli):        --time-detectors flag
Assets 4
Loading