Просмотр исходного кода

fix: reduce validator false positives — only fail on hard contradictions

Validator was rejecting chapters for minor issues like slightly
ahead-of-text inferences and incomplete hook tracking. Now only
fails for direct factual contradictions. Soft issues remain as
warnings with passed=true.

Discovered via autoresearch bench: 3/5 books failed chapter 1 due
to overly strict validation on reasonable settler extrapolations.
Ma 7 месяцев назад
Родитель
Сommit
4aea804371
1 измененных файлов с 8 добавлено и 1 удалено
  1. 8 1
      packages/core/src/agents/state-validator.ts

+ 8 - 1
packages/core/src/agents/state-validator.ts

@@ -61,7 +61,14 @@ Output JSON:
 }
 
 passed = true means no serious contradictions found. Minor observations are still reported as warnings.
-If there are no issues at all, return {"warnings": [], "passed": true}.`;
+If there are no issues at all, return {"warnings": [], "passed": true}.
+
+IMPORTANT: Set passed=false ONLY for hard contradictions — facts that directly conflict with the chapter text (e.g., character said to be dead but is alive in text, location stated as X but text clearly shows Y). Do NOT fail for:
+- Slightly ahead-of-text inferences (e.g., "about to go to X" when text shows intent but not arrival)
+- Missing details (state card doesn't capture every minor event)
+- Reasonable extrapolations from text (e.g., "slightly relieved" from "took a sip of water")
+- Hook management differences (hooks added/removed that don't contradict text)
+These should be warnings with passed=true, not failures.`;
 
     const userPrompt = `Chapter ${chapterNumber} validation: