DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to turn your coding chats into a fine-tuning dataset with AI Data Extractor

Extract, secret-scan, and load your assistant conversations as a training set for a chat model.

Credentials. Training on unscanned chats can teach a model to repeat API keys or other secrets that appear in code_context or tool_use.

Reverse engineering. Chat records can include proprietary code from employers or clients, so confirm you have the rights to use that data for training.

Who it is forAn ML engineer who wants to fine-tune a model on their own coding conversations and needs a clean, filtered dataset.

The problemRaw chat exports contain secrets, proprietary code, and personal file paths that must be removed before training.

The outcomeA filtered Hugging Face dataset of conversations that contain at least one assistant message, ready for chat-template formatting.

What you need first

  • Python 3.9 or newer installed on the machine that holds the chat history
  • The AI coding tools installed under your own user account, with local chat history present
  • Permission to read and store your own local data, and a secure place to keep the output
  • For the fine-tuning case: the Hugging Face datasets library and a tokenizer for your target model

Steps

  1. Run python extract.py --sources cursor,claude_code,aider to extract only the sources you want to train on.
  2. Run pip install detect-secrets --break-system-packages, then detect-secrets scan extracted_data/*.jsonl.
  3. Review code_context and tool_use fields for proprietary code, API keys, and personal file paths, and remove those records.
  4. In Python, load the cleaned files with load_dataset("json", data_files="extracted_data/*.jsonl", split="train") from the datasets library, then apply the README filter for assistant messages. (inferred, check the README)
  5. Format each example with tokenizer.apply_chat_template, after defining the tokenizer for your target model.

Prompt for your agent

Using the repo jatechan/ai-data-extractor, extract the sources cursor and claude_code, run detect-secrets on the output, and load the JSONL files in <project folder>/extracted_data with the datasets library. Report how many conversations remain after filtering for assistant messages. Verify that no record still contains a flagged secret before using it for training.

Caveat. The README example calls tokenizer.apply_chat_template without showing how to define the tokenizer, so you must supply it yourself.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page