TinyStories 15M with Lua grammar constraints and PyTorch inference.
File tools use a fresh virtual workspace. Writes never touch server files. With the gate off, virtual writes run automatically.
Select a model and task, then click Run experiment.
Execution trace
Evaluation dataset
These are measured runs from the bundled snapshot of lua-agent-evals. Download its JSONL files for prompts, expected outcomes and complete traces. Live visitor runs stay out of the published dataset.
Dataset snapshot: 4b60bb31ae40
The original result
30/30 valid under constraints. 0/30 free. 0/30 useful tool choices in both arms. This is the historical pure-Lua TinyStories measurement. Only an aggregate report and sample outputs survive; the dataset preserves that report separately. New TinyStories runs use PyTorch CPU kernels and retain every turn.
Two different questions
Did the output parse? Tool-call validity is parsed calls divided by generated turns. answer counts as a tool. A wrong tool name or wrong arguments can still parse.
Was the choice useful? Correct first tool checks one decision against the preset's accepted tools. Custom tasks show N/A. Task checks use the repository's small task-specific judges; these are not a general measure of model intelligence.
What the switches change
Grammar masks illegal tokens for TinyStories, or supplies a JSON schema to llama.cpp for Qwen. Free decoding gets the same prompt without a grammar. TinyStories scaffolds both THOUGHT: and CALL:; Qwen scaffolds CALL: in both arms. A constrained TinyStories call may be completed by the grammar at its token ceiling; each such case is counted in the run record.
The context slider changes the Lua prompt budget. The approval gate controls virtual writes; the adjacent checkbox supplies its decision. The loop detector stops repeated identical calls from executing. Error recovery feeds parse and tool errors back into the next step; switching it off ends the run at the first such error.
Models and scripted replay
TinyStories and Qwen run real checkpoints. On ZeroGPU hosting, TinyStories uses a shared GPU; local and Docker runs use CPU. Qwen runs on CPU. Scripted replay feeds a fixed sequence through the real Lua agent to isolate loop behavior. It does not adapt its answer when you remove a mechanism; the task judge detects the missing behavior. The grammar switch is disabled in this mode.
The Lua agent, parser, context manager and tool registry are the repository's implementation. TinyStories uses its original Lua grammar and tokenizer with PyTorch numerical kernels. Qwen uses official Q4_K_M weights and llama.cpp's schema grammar. Floating-point kernels, CPU versus GPU arithmetic, quantization and different grammars mean these are distinct experiments. Published evaluation rows were collected on CPU.
Context counts are exact BPE counts for TinyStories, Qwen token counts excluding chat formatting, and estimated bytes/4 for scripted tests. A budget or backend failure is shown as a stop reason, never counted as a malformed generated call.
jeorgexyz · MIT license