DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to rewrite extracted text files with look-alikes folded with Cjkfold

Replace look-alike glyphs in your text files directly, after confirming what will change.

Who it is forA developer or analyst with a set of extracted Chinese text files who wants clean copies ready for search or comparison.

The problemLook-alike glyphs look right on screen but break equality checks, dictionary lookups and deduplication.

The outcomeText files where the look-alike glyphs are replaced by the characters they imitate, with a report of what changed.

What you need first

  • Python 3.9 or newer with pip
  • Plain text extracted from the PDF, for example with pdftotext
  • A terminal for the command line examples, or a Python project for the library calls

Steps

  1. pip install cjkfold
  2. cjkfold check extracted.txt to see what would change.
  3. Copy the file to a backup location before rewriting it.
  4. cjkfold fold -w extracted.txt to rewrite the file in place. (inferred, check the README)
  5. cjkfold fold extracted.txt to print the folded text to the terminal for a quick look if you do not want to change the file. (inferred, check the README)

Prompt for your agent

Using the cjkfold repo, install it with pip install cjkfold, back up <path-to-text-file>, then run `cjkfold fold -w <path-to-text-file>`. Report the number of characters that changed by comparing the file before and after. Verify with `cjkfold check <path-to-text-file>` that it exits with code 0.

Caveat. The in-place rewrite changes the original file, and the README does not describe an undo, so keep a backup first.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page