DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to compare refusal-removal methods on one eval set with Phantom KV

Line up weight edits, activation hooks, and cache grafts on the same refusal metrics.

Authorization notice. Phantom KV is dual-use. Only use it on software, sites and systems you own or have explicit permission to test.

Reverse engineering. The evaluation involves prompts and completions that show refusal behavior, so keep harmful completions inside the review and do not publish them.

Who it is forAn AI safety reviewer at a lab who must decide how to test refusal robustness of an open model and needs a fair comparison of available methods.

The problemRefusal-removal methods report results with different setups, so their numbers are hard to compare directly.

The outcomeA side-by-side table of harmful refusals, harmless refusals, and KL divergence for the base model and each method arm, using one prompt suite and one decoding setting.

What you need first

  • Python environment and a Hugging Face causal language model; the README excerpt does not list hardware or dependency versions
  • Access to the GitHub repo lordx64/phantom-kv and its docs/TECHNIQUE.md file
  • A written evaluation plan that keeps harmful-prompt completions inside the review

Steps

  1. Read the comparison table in the README that contrasts heretic, weightless / GLP, and Phantom KV on permanence, runtime cost, and reversibility.
  2. Read the Results scoreboard and the methodology in docs/TECHNIQUE.md, noting the 60-prompt harmful suite, the 20-prompt harmless suite, greedy decoding, and teacher-forced KL against the base model's completions.
  3. Clone the repository with git clone https://github.com/lordx64/phantom-kv. (inferred, check the README)
  4. Run your own harness on the base model first and check that its counts match the base row (25/60 harmful refusals, 0/20 harmless refusals for Qwen3-4B-Instruct-2507).
  5. Run the same harness on each method arm you are comparing and record harmful refusals, harmless refusals, and KL mean and max for each row.

Prompt for your agent

In the repo lordx64/phantom-kv, read the Results scoreboard and the methodology in docs/TECHNIQUE.md, then reproduce the base row for <target model> on the 60-prompt harmful and 20-prompt harmless suites with greedy decoding. Expected output: harmful and harmless refusal counts plus KL mean and max for the base model. Verify that the base counts match the README row for Qwen3-4B-Instruct-2507 before reporting any other arm.

Caveat. The README says the project is deployed so far only on Qwen3 dense models, so results on other architectures are not yet confirmed.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page