DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to run the LoCoMo benchmark on a memory system with Voice Mem

Score Voice Mem on the LoCoMo dataset and check that the parser reads the data correctly first.

Who it is forA researcher or engineer who needs a reproducible score for a memory system and wants to compare it with published numbers.

The problemMemory benchmarks are easy to run wrong, for example by giving the reply model the full conversation instead of only retrieved memory.

The outcomeA score report with accuracy by question type and median retrieval latency, produced by a documented command.

What you need first

  • Python 3.10 or newer
  • An OpenAI API key, used for fact extraction when memories are written
  • Git and access to GitHub and Hugging Face to clone the repo and download the default models
  • Audio input (a WAV file or a microphone) if you want to ingest speech

Steps

  1. Clone the repo and run python evaluation/run.py --dataset locomo --data evaluation/examples/locomo_sample.json as a small check. (inferred, check the README)
  2. Run python evaluation/run.py --dataset locomo --data data/locomo.json --inspect to confirm the dataset parses without calling any model. (inferred, check the README)
  3. Set export OPENAI_API_KEY=sk-... and run python evaluation/run.py --dataset locomo --data data/locomo.json for the full benchmark. (inferred, check the README)
  4. Read evaluation/README.md for how to add another benchmark with one file and two functions.

Prompt for your agent

In the VoiceMem repo at <repo path>, run the LoCoMo sample with evaluation/run.py, then run the full dataset only after the sample passes. Report the overall score, the per-category scores, and median retrieval latency. Verify that the answer model received retrieved memory and not raw conversation history.

Caveat. The full run calls the OpenAI API, so the README's inspect mode should be used first to avoid unexpected charges.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page