# AI use cases for Cjkfold (cdcailf/cjkfold) Repo: [cdcailf/cjkfold](https://github.com/cdcailf/cjkfold) | Use cases: 4 | Page: https://dedupedai.com/use-cases/cdcailf-cjkfold/ A Python library and command line tool that folds look-alike CJK radical glyphs back to the ordinary Han characters they imitate, and reports look-alike and invisible characters in text. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Check PDF-extracted Chinese text for look-alike glyphs: https://dedupedai.com/use-cases/cdcailf-cjkfold/check-pdf-text-for-lookalikes/ - Fold a RAG corpus and its queries so Chinese terms match: https://dedupedai.com/use-cases/cdcailf-cjkfold/fold-rag-corpus-and-queries/ - Audit a folder of extracted text for hidden characters: https://dedupedai.com/use-cases/cdcailf-cjkfold/scan-corpus-for-invisible-characters/ - Rewrite extracted text files with look-alikes folded: https://dedupedai.com/use-cases/cdcailf-cjkfold/fold-extracted-text-in-place/ ## How to check PDF-extracted Chinese text for look-alike glyphs with Cjkfold Repo: [cdcailf/cjkfold](https://github.com/cdcailf/cjkfold) | Difficulty: Intermediate | Time: 15 minutes | Basis: From the README Find Kangxi and radical look-alike characters in extracted text and fail an ingest run on them. **Who it is for:** A developer whose PDF ingest pipeline returns missed keyword matches on Chinese documents and needs to find out why. **The problem:** Look-alike characters display correctly but do not compare equal to the ordinary characters, so searches miss with no error. **The outcome:** A report that lists each look-alike with its line, column and ordinary replacement, plus an exit code a pipeline can act on. ### Steps 1. `pip install cjkfold` 2. Extract the PDF to plain text with a tool such as pdftotext, and save the result as extracted.txt. 3. `cjkfold check extracted.txt` 4. Read the report. Each line gives the position, the look-alike character and the character it renders as. 5. `cjkfold check --quiet docs/*.txt` in the ingest pipeline, so a non-zero exit stops bad files before they reach the index. ### Prompt to paste into your agent ```text Using the cjkfold repo, run `pip install cjkfold`, then run `cjkfold check `. Report the number of look-alike characters and list the first three with their line and column. Verify by running `cjkfold fold -w` on a copy of the file and then `cjkfold check` on that copy, which should exit with code 0. ``` ### Caveat The check command only looks for the look-alike family, so a file full of zero-width spaces will still report clean and needs the scan command as well. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/cdcailf-cjkfold/check-pdf-text-for-lookalikes/ ## How to fold a RAG corpus and its queries so Chinese terms match with Cjkfold Repo: [cdcailf/cjkfold](https://github.com/cdcailf/cjkfold) | Difficulty: Advanced | Time: 1 hour | Basis: From the README Apply folding on the way into the index and on the way out, then restore verbatim matches with rerank. **Who it is for:** An engineer running a retrieval pipeline over Chinese PDFs whose vector search ranks unrelated short documents above the one that contains the query verbatim. **The problem:** Folding only the index or only the query silently disables matching, and dense retrieval handles literal tokens poorly. **The outcome:** A pipeline where the corpus and the query are folded the same way, and the verbatim matches are moved back to the top of the results. ### Steps 1. `pip install cjkfold` 2. `from cjkfold import fold, fold_documents, rerank` 3. Wrap the loader output with `documents = fold_documents(loader.load())` before embedding. 4. Fold the question with `hits = vectorstore.similarity_search_with_score(fold(question), k=20)`, then pass the results through `rerank(question, documents, scores)`. 5. Re-embed the existing index, since text already stored is not repaired retroactively. ### Prompt to paste into your agent ```text Using the cjkfold repo, add fold_documents to the document loading step of and fold the question before the vector search. Report which documents move up after rerank for the query . Verify that the folded question and the folded stored text both contain the target term when checked with the in operator. ``` ### Caveat Documents already in your vector store keep their look-alike characters until you rebuild the embeddings, so the fix only applies to text added or re-indexed after the change. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/cdcailf-cjkfold/fold-rag-corpus-and-queries/ ## How to audit a folder of extracted text for hidden characters with Cjkfold Repo: [cdcailf/cjkfold](https://github.com/cdcailf/cjkfold) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README Scan a whole folder for look-alikes, compatibility characters and invisible characters in one report. **Who it is for:** A data engineer who has inherited a folder of extracted text and needs to know what kinds of hidden problems it contains. **The problem:** Zero-width spaces leave no visible glyph, so two strings differing only by one look identical and compare unequal. **The outcome:** A per-file report that groups findings by class and says what can be fixed by folding, by NFKC, by stripping or only by hand. ### Steps 1. `pip install cjkfold` 2. `cjkfold scan corpus/` 3. Read the grouped findings. Treat `manual` items, such as zero-width joiners in emoji, as items to review rather than delete. 4. `cjkfold scan --json corpus/` if you want one JSON object per file for further processing. (inferred, check the README) 5. `cjkfold scan --strict corpus/` if you want the run to exit 1 when a foldable or compatibility character is found. (inferred, check the README) ### Prompt to paste into your agent ```text Using the cjkfold repo, run `cjkfold scan --json ` after installing it with pip install cjkfold. Report the number of files scanned, the number of findings per class, and the files with the most invisible characters. Verify that scan exits with code 0 when it only reports findings, as the README describes. ``` ### Caveat The scan command exits 0 even when it finds problems, so a pipeline needs --strict or a separate check to fail on findings. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/cdcailf-cjkfold/scan-corpus-for-invisible-characters/ ## How to rewrite extracted text files with look-alikes folded with Cjkfold Repo: [cdcailf/cjkfold](https://github.com/cdcailf/cjkfold) | Difficulty: Intermediate | Time: 15 minutes | Basis: From the README Replace look-alike glyphs in your text files directly, after confirming what will change. **Who it is for:** A developer or analyst with a set of extracted Chinese text files who wants clean copies ready for search or comparison. **The problem:** Look-alike glyphs look right on screen but break equality checks, dictionary lookups and deduplication. **The outcome:** Text files where the look-alike glyphs are replaced by the characters they imitate, with a report of what changed. ### Steps 1. `pip install cjkfold` 2. `cjkfold check extracted.txt` to see what would change. 3. Copy the file to a backup location before rewriting it. 4. `cjkfold fold -w extracted.txt` to rewrite the file in place. (inferred, check the README) 5. `cjkfold fold extracted.txt` to print the folded text to the terminal for a quick look if you do not want to change the file. (inferred, check the README) ### Prompt to paste into your agent ```text Using the cjkfold repo, install it with pip install cjkfold, back up , then run `cjkfold fold -w `. Report the number of characters that changed by comparing the file before and after. Verify with `cjkfold check ` that it exits with code 0. ``` ### Caveat The in-place rewrite changes the original file, and the README does not describe an undo, so keep a backup first. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/cdcailf-cjkfold/fold-extracted-text-in-place/