DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to compare full Jev Mem against ablated controllers on LoCoMo with Jev Mem

Run LoCoMo questions with full, write-ablated, and read-ablated Jev Mem to see which control layer changes answers.

Credentials. Benchmark runs call the answer and judge models with your keys, so keep the .env file local and do not share keys in agent logs.

Who it is forA researcher reproducing or extending the Jev Mem paper who needs to measure the effect of write control and read control separately.

The problemA single score does not show whether write control or read control drives the gains, so each has to be tested alone.

The outcomeScored LoCoMo question results for the full configuration and for each ablation, with timing and retrieval traces recorded.

What you need first

  • Python 3.11 or newer and a terminal
  • A local copy of the repository (the README assumes you run commands from the repository root)
  • For live runs: a TypeSafe API key and an OpenAI API key, plus network access, since live runs may incur charges
  • For benchmarks: the LoCoMo dataset placed at data/locomo10.json, obtained from its original distributor per the data setup guide

Steps

  1. Place the LoCoMo dataset at data/locomo10.json by following the data setup guide.
  2. Run the full configuration with python -m jev_mem.benchmarks.locomo --dataset data/locomo10.json --jev-config config/jev_mem.json --sample 0 --model gpt-4o-mini --max-questions 10 --category-to-test 1,2,3,4 --best-of-n 1. (inferred, check the README)
  3. Run the write ablation by adding --no-jev-write to the same command, and the read ablation by adding --no-jev-read, using a separate --cache-dir PATH for each run. (inferred, check the README)
  4. Compare the scored outputs and the recorded timing and decision traces across the runs.

Prompt for your agent

In the Jev-Mem repository at <target-folder>, run the LoCoMo benchmark on data/locomo10.json for sample 0 with 10 questions, once with the full config and once each with --no-jev-write and --no-jev-read, using separate --cache-dir paths. Report the three overall scores and confirm that all three runs used the same dataset subset.

Caveat. The README says these commands are starting points and that their subsets and scoring settings do not reproduce the full paper evaluation by themselves.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page