Scholaryx

01Overview

Scholaryx runs four LLM components in a fixed order — a search agent, a reader agent, a writer chain and a critic chain — to produce a short research report with sources and a scored critique. A web workbench shows each stage as it runs, the intermediate outputs, which cited links actually appear in the gathered material, and the measured token usage and cost.

The pipeline began from a public tutorial. It was extended with the workbench and its API, per-stage tracking, measured token usage, a citation cross-check, usage limits and server-side request forgery (SSRF) protection for the public deployment, pinned dependencies and automated tests.

02Problem statement

Given a free-text topic, gather recent material from the web, read at least one source in depth, and write a report with an introduction, at least three key findings, a conclusion and a list of sources — while making each step’s inputs and outputs visible so a reader can check where claims came from.

03Motivation

Splitting the task into components with one responsibility each makes every step inspectable: the search results, the page that was read, the draft and the critique can be examined and tested separately. The design is deliberately sequential — there is no planner, no parallelism and no feedback from the critic back to the writer — which keeps runs predictable and their cost easy to bound.

04System architecture

1. Search agent — a LangChain tool-calling agent (gpt-4o-mini) with one tool, web_search: Tavily, 5 results, each trimmed to title, URL and a 300-character snippet.
2. Reader agent — a second agent receives the first 800 characters of the search output, picks the most relevant URL and calls scrape_url: requests + BeautifulSoup, 8-second timeout, first 3,000 characters of cleaned text.
3. Writer chain — a prompt → LLM → parser chain that writes the report from the search results and the scraped text, with required sections: Introduction, Key Findings (at least three), Conclusion, Sources.
4. Critic chain — a separate prompt asks the same model for Score: X/10, strengths, areas to improve and a one-line verdict.
5. Server — FastAPI runs each request in a worker thread, derives live stage status from the pipeline’s own progress markers, and records token usage with LangChain’s OpenAI callback.

05Technical implementation

• All model calls use gpt-4o-mini at temperature 0. API keys are read only from server environment variables and are redacted from run logs; the browser never receives them.
• Usage limits for the public demo: at most 20 runs per day overall and 3 per visitor per hour, one run at a time; limited requests receive HTTP 429 with a retry time. Other visitors’ runs are not listed.
• SSRF protection in the scraper: only http(s) URLs; every address the host resolves to must be public (loopback, private, link-local, cloud-metadata and shared ranges are refused, including numeric and IPv4-mapped forms); redirects are followed manually, re-checked, and limited to three; responses are capped at 2 MB.
• Dependencies are pinned to the tested versions. Eleven automated tests cover the scraper guard and the usage limits with a fake pipeline, so they make no paid calls.

06Agent design

Only the first two components are agents in the tool-calling sense; the writer and critic are fixed prompt chains. The reader is given a short slice of the search output rather than all of it, which keeps its prompt small but means it can only choose among the first few results. Each component’s output is passed forward as plain text, so the whole run can be reconstructed from the stored intermediate outputs.

07Evaluation methodology

There is no reference dataset for open-ended research reports, so evaluation is limited to what can be checked per run:

• Structure: whether the required sections are present and the report has at least three findings.
• Citation cross-check: whether each URL cited in the report appears in the search or reader output (catches invented links; does not verify that the cited page supports the claim).
• Critic score: recorded, but it is the same model grading its own pipeline, so it is not treated as a quality measure.
• Cost and latency: per-stage time, tokens and cost from the callback.

08Verified results

Protection checks

• 11 / 11 automated tests pass: 17 blocked URL forms are never requested, a redirect to a private address is stopped, the redirect and size limits hold, a real public page returns the same text as the original scraper, and the hourly and daily limits behave as specified.
• On the live site, forged X-Forwarded-For and True-Client-IP headers did not create a fresh per-visitor allowance, and a forged CF-Connecting-IP header was rejected by the hosting edge.

Two runs on one topic are a demonstration, not a benchmark: they show the pipeline works end to end and what it costs, not how good its reports are in general.

09Demonstration video

1:16Play Scholaryx demo, 1 minute 16 seconds

Run 2 above, recorded in real time on the live deployment; no audio. The stages progress live, then the report, the critic’s evaluation, the five sources and the intermediate outputs are shown; the end card lists the run’s measured numbers.

10Deployment and source code

• Live demo: anshaiml369.github.io/scholaryx — Hugging Face Space (Docker). API keys are stored as private Space secrets, never in the code.
• External services: OpenAI (gpt-4o-mini) and Tavily; every run spends credit, which is why the demo is limited.
• Source code: available on request.

11Limitations and known failure modes

• Self-grading. The critic is the same model judging its own pipeline without a reference answer; its score is not calibrated.
• Narrow reading. The reader sees only the first 800 characters of search output and reads one page, truncated to 3,000 characters.
• Citations are checked for presence, not support. A cited URL can appear in the gathered material while the claim attached to it is wrong.
• Untrusted page content. Scraped text is passed to the writer as-is, so instructions embedded in a web page could influence the report (prompt injection is not mitigated).
• Limits reset on restart. Usage counters live in memory; a provider-side spending limit is the hard cap. A host that changes its address between the check and the fetch (DNS rebinding) is not fully covered.
• Results appear only when the whole run finishes, and run history is held in memory on the server.

12Future improvements

• A small evaluation set of topics with reference facts, scored independently of the pipeline’s own model.
• Claim-level citation checking: does the cited passage support the sentence that cites it?
• Reading several sources in parallel, and treating scraped text as untrusted input.
• Streaming partial outputs and persisting runs.