Logic Drift Detector

A CLI tool that queries an LLM about a knowledge graph or JSON schema and uses an SMT solver to flag logical inconsistencies in the response.

Difficulty: weekend | Stack: Python, NetworkX, Z3 (Microsoft SMT solver), Anthropic SDK, Typer CLI

Who this is for

Developers building RAG pipelines over structured data (KGs, databases, ontologies) who need to catch hallucinated or self-contradictory LLM outputs before they reach users.

Build steps

1. Load a small domain knowledge graph (e.g., a film DB or medical ontology) into NetworkX and serialize a subset as JSON-LD for context injection.
2. Prompt the LLM to answer multi-hop questions about the graph (e.g., ‘Who directed films starring both X and Y?’) and capture the full response.
3. Parse the LLM’s answer into a set of logical propositions using a lightweight regex + heuristic extractor.
4. Encode those propositions as Z3 constraints alongside the ground-truth graph facts and call solver.check() to detect contradictions.
5. Surface failing constraints as human-readable ‘drift reports’ in the CLI, showing which claim violated which graph fact.

Risks

• Proposition extraction from free-form LLM text is brittle — complex sentences may parse into ambiguous or wrong logical forms, causing false positives.
• Z3 constraint encoding requires manual mapping from domain concepts to sorts/relations, which becomes a significant modeling effort for larger schemas.
• The KG subset injected into context may be too large for the model’s window, forcing chunking strategies that themselves introduce inconsistency.