STUDY WITH ALFONS

I’m starting to be afraid of my project, and of myself. I just wanted to write a little tool that’s less annoying than Google translate, or ChatGPT, or DeepSeek, when it comes to transcribing Chinese characters into Hànyǔ Pīnyīn. I just didn’t want to see the same rookie mistakes made over and over again, stubbornly made without any hope for improvement, not any time soon anyway.

So I sat down to write this little tool. And with Grok xAi, and a bit of ChatGPT, and a couple of amazing open source libraries (such as rakutenMA and LibreTranslate) it’s actually quite convenient to lay down a bit of code.

But, alas. Lo-and-behold. What a monster I have created! Hundreds of hours of work, thousands of lines of code. Have a look at a snapshot of the user interface:

This is how it looks now. You drop your text written in Chinese characters in the top left, and get Hànyǔ Pīnyīn on the right, and all the tools you need to manipulate and re-arrange the final text. Terribly beautiful. A complex task made ease.

There’s still a few quirks, and quite a few rules to implement and to improve, to make it (almost) compliant with GB/T 16159-2012 — but I already love to use it, and love to use it over anything else.

However… it’s a lost cause from the start: modern text segmentation and tokenisation tools operate with 70 to 98% accuracy. Some sentences come out better than others, before I try to fix the mistakes. But it will never be 100%.

And some years later, when Baidu, Alibaba and Google, etc, finally jump on the Pīnyīn-train, my software will be obsolete. Yet, I started this, fully aware of its transitoriness. A little bit crazy. But beautiful.

O. M. G. – in my obsessive spree (that rhymes!) I moved on to create yet another tool in the field of Hànyǔ Pīnyīn! Now I have like 7 projects parallel in the cooking, er, kitchen… in the making! …is what I’m saying!

This one was a really tuff cookie. I had many problems to solve. It took me the better part of the entire month. Next to sleeping – and a daily short session of self-Feldenkrais for self-preservation – there was not much else I did. But here I am, here we are!

My transcription tool, even in its early prototype stage, is already on par with Google translate, ChatGPT and Deepseek… Deepseek! of all large language models, in terms of transcription quality — which is not very difficult because they are very sloppy in this regard; nobody seems to care about Pīnyīn orthography, not even the machinese.

I use the Made-In-Japan rakutenMA for segmentation (which works in any browser and is purely javascript based), paired with some simple lookups from creative commons dictionary CC-CEDICT (which is second by a wide margin to the great ABC-Dictionary by Hawaii Press, but it’s all we have for now, and for that I’m grateful), a frequency based table for single character Chinese morphemes, and a smooth working user interface (overall design still needs to be created.)

Next, I will need to focus on my actual work as a Feldenkrais teacher, for the last week of the month, and produce two Feldenkrais videos. Despite my crazy work hours due to my software projects, I still need to pay the rent. The upcoming two videos will probably be the slowest and deepest I’ve produced so far in the past 15 years; I hope to be able to get closer to live-class experience in my videos.

For the theme of the upcoming video, I’ve made my pick in the beginning of the month already (dealing with lingering shoulder pain due to sitting in front of the computer too much). Looking forward to teaching and filming!

And then, next month, I will start implementing the official government rules (GB/T 16159-2012) for Hànyǔ Pīnyīn into my new transcription software, making it the best in the world, second to none!

Also, I’m planing to wrap my colorful Pīnyīn Colors app into a MacOS app, or maybe even an iPad app, register and pay for an Apple developer account (walk of shame), and put it on the Apple App-Store (for free, as most of my work is, so is my plan.)

Ok, now, no time to proof-read, ok, but just one run through, no time Toulouse, I need to get on with it!

What a crazy past 2-3 months! Almost every day I was working for up to 10 or even 14 hours on my projects about Hànyǔ Pīnyīn, the official romanization system for Standard Mandarin Chinese. I’ve produced quite a few projects:

• it started with a grammar book, which then lead to
• several apps,
• several typing and input helpers,
• including a unique, highly innovative typing helper to write in Pīnyīn without numbers and selection boxes,
• a dictionary filter and dictionary interface,
• I even put Pīnyīn diacritics on a font,
• and worked on several algorithms.

Algorithms, algorithms everywhere

For example, my algorithm for coloring vowels (or vowel groups) in HTML containers: what started as a lagging, heavy, cumbersome-to-use prototype (as shown in a previous post), is now a blazingly fast algorithm that can handle tens of thousands of words, on large canvas.

As a surprising side-effect, this unleashed the potential to play with Chinese words and text in an artistic way, in a way hardly ever done before, because the means simply didn’t exist before.

Here’s a list of all 8-letter Chinese words that contain syllables (or words) that sound like »yào« — or at least, all the words listed in the CC-CEDICT dictionary. Sorted with PinyinAbcSort, my algorithm that can sort Hànyǔ Pīnyīn (fast).

This must be the Austrian Viennese blood boiling inside of me, this innovative, unconventional play with language and words, a genetic feature that suddenly, unexpectedly, rose up to get hold of me and my senses.

And I can’t stop. It’s like writing a novel. The story unfolds. The story pushes me, the writer, forwards, manically, unceasingly, like in a fever dream. But instead of plots and chapters, I produce product designs and prototypes.

But to what ends? Where does this lead to? This is becoming far too big for me.

Today I’ve spent almost the entire day, from 7:30 am to 22:20 pm, working on my coloring of vowels. Fun. Borderline crazy. Total obsession. But I guess that’s the only way to get things done.

And I have to say, it’s b-e-a-u-t-i-f-u-l ! A total beauty. I love it.

The most difficut part was to get the text to copy. That’s probably something anyone is taking for granted. Copy, Paste, Print. But to get “copy the whole page into the clipboard” to work, that alone did cost me around 8 hours, and I had to leave it at copying the whole page. I tried a many ways, and also a few well-established 3rd party solutions (such as highlightjs and quilljs) but to no avail. At times I felt a bit like Bilbo in the Mirkwood – with some Elves somewhere having a picknick, singing merry tunes all along.

On the upside, it now copies nicely into rich text apps like Apple Pages and TextEdit – and maybe even Microsoft Word (who knows?) – and there the colored text can be printed or edited further.

You can play with Pīnyīn Tone Vowel Coloring here: alfonsgrabher.com/pinyin-colors

“While Chinese writing is traditionally understood as logographic and syllable-based, Hànyǔ Pīnyīn is written using the Latin alphabet, letter by letter.” — Alfons Grabher

If this fact is taken seriously, it opens up many new possibilities and areas for creative expression. For example, by casting a fresh light on tone coloring as a tool for education and language study.

In Mandarin, tones are pitch patterns realized on vowels. The vowel is where the tone truly “lives,” as it sustains the sound long enough for the pitch to rise, fall, dip, or remain level. Consonants, by contrast, are typically too short or voiceless to carry pitch and play little to no role in tone perception.

Since tones are phonetically manifested on vowels, it seems reasonable — in an experimental or pedagogical context — to apply visual tone indicators, such as color, specifically to vowels rather than to entire syllables. This approach emphasizes the part of the syllable that actually conveys tonal information and may enhance learners’ sensitivity to tone production and perception.

I played with these ideas focusing on the pedagogical and perceptual side – how learners can visually and intuitively grasp tones. To show how this might look in practice, I put together three visually distinct examples, using Pleco’s standard tone colors for simplicity:

Style 1: Colored vowels inside soft colored rectangles

Style 2: Colored vowels with a soft glow

Style 3: Vowels inside large, colored rectangles

To my mind, this is clean, intuitively readable, and – dare I say – pedagogically brilliant. Simply by coloring the vowels, we visually encode the essence of tone in a way that’s instantly graspable and clearly highlights the tonal components of each word.

Getting the pronunciation and tones right is one of the primary goals for beginner learners of Mandarin. Coloring – and thereby highlighting – the vowels, which carry the tones, feels almost revolutionary. Here are two arguments in favor of what I call “Pīnyīn Vowel Tone Mapping”, or “Pīnyīn Vowel Tone Coloring” (still looking for a name):

Vowel-centric highlighting

We’re staying faithful to the phonetic reality: tones live on the vowel, and we’re giving that fact a visual identity.

Color-coded tonal categories

With a little bit of practice, vowel colors provide an instant way to relate to tone, helping with tone perception and possibly even improving the ability to remember tones.

To conclude this post, instead of just presenting static example text, I’ve turned this idea into an interactive experience. You can try it out here:

Link: alfonsgrabher.com/pinyin-colors

Differences in word order between PinyinAbcSort and the ABC Chinese-English Dictionary

Link to the Github repo: http://github.com/alfons/PinyinAbcSort

In the esteemed ABC Chinese–English Comprehensive Dictionary published by the University of Hawai‘i Press, word order follows a Western alphabetical principle, quite strictly so: entries are sorted primarily by the base letters of the Latin alphabet, with Pīnyīn diacritics merely considered as tie-breakers when the base spellings are otherwise identical.

By contrast, the PinyinAbcSort-algorithm sorts words by the base letters of the Latin alphabet while fully respecting each diacritic as a core feature—allowing tones to shape the order from the very beginning, rather than serving merely as secondary tie-breakers. As a result, the sorting is quite different. For example, a side-by-side comparison:

2. Philosophy of Word List Design

Alphabetization isn’t neutral—it’s a design decision. Languages which use an extended Latin alphabet generally have their own conventions for treatment of the extra letters. Here, too, one road prioritizes ease of lookup for users familiar with English language dictionaries; the other preserves phonological fidelity by respecting the tone marks of Chinese.

• The ABC dictionary approach is Westernized: prioritizing base-letter order à la English.
• The PinyinAbcSort-algorithm approach reflects a more Sinophone-conscious logic, aligning with how tones carry semantic weight in Mandarin.

This distinction touches on:

• Does the order reflect how Chinese is actually pronounced?
• Should sorting reflect technical accuracy or acedemic tradition?
• What’s easier or more natural to search for?

3. Personal Commentary

Language is highly personal. The way we speak is part of our identity. Therefore, I don’t think a choice of roads can be made by mere academic reasoning. Here is my personal commentary on why I designed PinyinAbcSort the way it is.

“Children learn what they live. Put kids in a class and they will live out their lives in an invisible cage, isolated from their chance at community; interrupt kids with bells and horns all the time and they will learn that nothing is important; force them to plead for the natural right to the toilet and they will become liars and toadies; ridicule them and they will retreat from human association; shame them and they will find a hundred ways to get even.” — John Taylor Gatto, in his book Dumbing Us Down.

My extreme, extraordinary, and uncompromising passion for Hànyǔ Pīnyīn does not stem from academic study, nor from an early acquaintance with the works of John DeFrancis, Zhōu Yǒuguāng, Yīn Bǐnyōng, and others. These figures are among the most prominent pioneers of Hànyǔ Pīnyīn, and without their vision and perseverance, it would not exist. But I only discovered their names, books and writings after my own path had already taken shape, after I had become deeply and irrevocably committed to the fruits of their labour, Hànyǔ Pīnyīn.

In June 2024, I’ve made a personal decision:

From hereafter I will use Hànyǔ Pīnyīn as a complete replacement for Chinese characters. For me this concerns diary-style writing and casual, leisure reading (novels, interview transcripts, subtitles, etc).

Where does my passion come from?

My passion comes from nearly two decades of being shamed, ridiculed, corrected, and condescended, as a “lǎowài”, a perpetual foreigner, deemed incapable of grasping one fundamental truth: that Mandarin has four tones.

I’ve spent thousands of hours, and thousands of dollars, trying to learn Mandarin in every imaginable setting. And yet, I failed spectacularly, miserably and almost completely, and nearly every teacher I’ve encountered did sing the same refrain:

“Your pronunciation is wrong. Chinese has four tones. You need to learn this first.”

Ironically, the very script they use for teaching Chinese, and rely on themselves — Chinese characters — offers little to no tonal guidance. In contrast, Hànyǔ Pīnyīn not only represents the four tones — it excels at doing so. And yet, most teachers I’ve met cannot read Hànyǔ Pīnyīn with ease. They shy away from it, downplay its significance, try to talk me out of it — even though it encodes tone with precision and elegance.

Anyone who has read the official standard (GB/T 16159–2012) or any major scholarly treatment of Hànyǔ Pīnyīn will know how much care is given to the spelling of names, how meticulously tone rules are applied in personal and place names. And yet, many of the very pioneers of Pīnyīn often spell their own names without diacritics. I find this unacceptable.

Chinese has four tones. This much I have learned. Hànyǔ Pīnyīn spells them out. This is my gospel.

The reason why “PinyinAbcSort” can’t be spelled “PīnyīnAbcSort”

While “PīnyīnAbcSort” would be linguistically accurate, it’s not fit for technical use. Diacritics can break URLs, aren’t allowed in most programming identifiers, and may cause issues in older or non-ASCII-safe systems. Sad, but acceptable—as an exception due to technical limitations. Confucius says: blind passion fades, passion that’s guided by reason endures.

Confucius also says: passion burns like fire, few add tone marks.