DEDUPED Hot AI repos, turned into use cases Get 5 a day

What you can do with Cjkfold

A Python library and command line tool that folds look-alike CJK radical glyphs back to the ordinary Han characters they imitate, and reports look-alike and invisible characters in text.

Trended earlier

cdcailf/cjkfold on GitHub

Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk.

What do you want to do?

4 use cases

  • Check PDF-extracted Chinese text for look-alike glyphs

    Find Kangxi and radical look-alike characters in extracted text and fail an ingest run on them.

    View prompt
    Using the cjkfold repo, run `pip install cjkfold`, then run `cjkfold check <path-to-extracted-text-file>`. Report the number of look-alike characters and list the first three with their line and column. Verify by running `cjkfold fold -w` on a copy of the file and then `cjkfold check` on that copy, which should exit with code 0.

  • Fold a RAG corpus and its queries so Chinese terms match

    Apply folding on the way into the index and on the way out, then restore verbatim matches with rerank.

    View prompt
    Using the cjkfold repo, add fold_documents to the document loading step of <path-to-pipeline-file> and fold the question before the vector search. Report which documents move up after rerank for the query <query-text>. Verify that the folded question and the folded stored text both contain the target term when checked with the in operator.

  • Audit a folder of extracted text for hidden characters

    Scan a whole folder for look-alikes, compatibility characters and invisible characters in one report.

    View prompt
    Using the cjkfold repo, run `cjkfold scan --json <path-to-corpus-folder>` after installing it with pip install cjkfold. Report the number of files scanned, the number of findings per class, and the files with the most invisible characters. Verify that scan exits with code 0 when it only reports findings, as the README describes.

  • Rewrite extracted text files with look-alikes folded

    Replace look-alike glyphs in your text files directly, after confirming what will change.

    View prompt
    Using the cjkfold repo, install it with pip install cjkfold, back up <path-to-text-file>, then run `cjkfold fold -w <path-to-text-file>`. Report the number of characters that changed by comparing the file before and after. Verify with `cjkfold check <path-to-text-file>` that it exits with code 0.

How to use Cjkfold

  1. Check you have what it needs: Python 3.9 or newer with pip; Plain text extracted from the PDF, for example with pdftotext; A terminal for the command line examples, or a Python project for the library calls.
  2. Open cdcailf/cjkfold on GitHub and read the README and license.
  3. pip install cjkfold
  4. Extract the PDF to plain text with a tool such as pdftotext, and save the result as extracted.txt.
  5. cjkfold check extracted.txt
  6. Read the report. Each line gives the position, the look-alike character and the character it renders as.

These steps come from our top use case, Check PDF-extracted Chinese text for look-alike glyphs, which also has a copy-paste agent prompt.

Cjkfold alternatives · Best AI repos for Software and DevOps

About this repo

  • Who it is forDevelopers and data engineers who ingest Chinese or other CJK text from PDFs and see searches, retrieval or deduplication quietly miss matches.
  • Why it is hot
  • LanguagePython
  • Sourcegithub.com/cdcailf/cjkfold · Back to Hot repos

Requirements

  • Python 3.9 or newer with pip
  • Plain text extracted from the PDF, for example with pdftotext
  • A terminal for the command line examples, or a Python project for the library calls
Maintain Cjkfold? Add the Featured on Deduped badge

Paste this into your README:

[![Featured on Deduped](https://dedupedai.com/badge/cdcailf-cjkfold.svg)](https://dedupedai.com/use-cases/cdcailf-cjkfold/?ref=badge)

More badge styles and HTML

Questions about Cjkfold

What is Cjkfold?

A Python library and command line tool that folds look-alike CJK radical glyphs back to the ordinary Han characters they imitate, and reports look-alike and invisible characters in text. It is an open-source project on GitHub: https://github.com/cdcailf/cjkfold.

Who is Cjkfold for?

Developers and data engineers who ingest Chinese or other CJK text from PDFs and see searches, retrieval or deduplication quietly miss matches.

What can I do with Cjkfold?

4 use cases so far, for example: Check PDF-extracted Chinese text for look-alike glyphs; Fold a RAG corpus and its queries so Chinese terms match; Audit a folder of extracted text for hidden characters. Each has steps and a copy-paste agent prompt.

What do I need to run Cjkfold?

Python 3.9 or newer with pip; Plain text extracted from the PDF, for example with pdftotext; A terminal for the command line examples, or a Python project for the library calls.

What are alternatives to Cjkfold?

Other hot AI repos in Software and DevOps: Harness2, iFixAi, Orca, Diagram Design, AI Memory.

Is Cjkfold safe to run?

Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.