How I Built an AI Research Skill That Checks Its Own Citations

Key Takeaways

• 55% of GPT‑3.5 citations fabricated; GPT‑4 improved to 18%
• Jean adds scoped searches, multi‑cycle research, and citation verification
• Skill pulls from Semantic Scholar, arXiv, then expands via citation trails
• Verified citations need DOI/URL; unverifiable sources are automatically removed

Pulse Analysis

AI‑driven text generators have long promised instant literature reviews, but the prevalence of fabricated citations has eroded trust. A 2023 study showed more than half of GPT‑3.5’s references were nonexistent, and even GPT‑4 left nearly one‑fifth of citations unverifiable. This hallucination problem forces researchers to spend disproportionate time cross‑checking sources, undermining productivity and raising the risk of propagating false findings across scholarly and corporate pipelines.

Jean tackles the issue by imposing a disciplined workflow on Claude Code. First, it scopes the query, defining depth and focus before any search begins. It then executes three research cycles: an initial sweep of core academic repositories, a targeted expansion to specific authors and journals, and a final citation‑trail crawl to capture foundational works. Each candidate paper is scored for relevance, authority, and recency, and must present a valid DOI or URL before being retained. Unverified entries are automatically discarded, ensuring the final report contains only traceable, peer‑reviewed sources.

The broader implication is a shift toward verifiable AI research assistants in both academia and industry. By embedding verification loops, tools like Jean can become standard components of corporate knowledge‑management systems, accelerating product development and strategic planning while mitigating misinformation risk. As large language models evolve, integrating transparent provenance checks will likely become a regulatory expectation, positioning early adopters to reap efficiency gains and maintain credibility in data‑driven decision making.