Translation Scripting for KDE

Extensions in this group allow you to reach some data existing internally within the scripting system, which might be usefull in some situations.

Unicode strings are not implemented as a special data type, they are just normal Guile strings in UTF8 format. That means that you can use all built-in Guile functions to handle Unicode strings, however any function which deals with specific characters in string may not give expected result.

For example, string-downcase should return string in lower case, but if there are any non English letters, they will not be lowercased; on the other hand, function string=? will always work, as it compares strings in global.

In short, whenever you are operating on pure English strings, feel free to use built-in string handling functions, but consider some of those described below when handling strings in your language.

As soon as one moves away from caseless language, like English, trouble with correct inflection of non literal/non numerical arguments of messages creeps in. Take following two messages from kgeography.po (placeholders stand for names of countries), translated into Serbian language:

#: divisioncapitalasker.cpp:32
msgid "The capital of %1 is..."
msgstr "Prestonica %1 je..."

#: mapasker.cpp:177
msgid "Please click on %1"
msgstr "Kliknite na %1"

In Serbian, which has several cases of nouns, and case endings are appended to the noun itself, these translations sound quite "illiterate". Both have nominative case of country name, although in translation of "The capital of %1 is..." it should be genitive case, and in translation of "Please click on %1" accusative case.

(To be quite fair, KGeography also defines expanded messages of this kind (like "The capital of Greece is..."), but that makes its PO file having over 4000 messages. So the scripted solution is still going to come in handy.)

From the scripting viewpoint, this is the same problem as described in introductory example, so you could solve it in the same way. But, in this case, we can also go for a more elegant solution. Note that in same PO file, you have country names defined separately, for example (more or less):

#: mapsdatatranslation.cpp:115
msgid "Greece"
msgstr "Grčka"

So, we know that by the time one of those problematic messages above comes about, the appropriate country translation will have been invoked already. This fact allows us to set needed forms dynamically in PO file itself, rather than statically in a scripting module.

With this approach in mind, we provide needed cases of country names by scripting their translations like this:

#: mapsdatatranslation.cpp:115
msgid "Greece"
msgstr "Grčka"
"|/|"
"$[set-form 'genitive \"Grčke\" 'accusative \"Grčku\"]"

Intention here is pretty obvious, but some words about the details. The whole script is one big interpolation using function set-form, which we must define in scripting module (but it will be a short one!) This function takes case-form pairs in turn, where case is a symbol (denoted by single quote in front), and forms are strings. Since inside interpolation is Scheme code, strings must have quotes, thus the quoting of forms. Since here we just want to set the forms, and not to mess with translation, set-form must also return ordinary translation itself.

The problematic messages can now be scripted like this:

#: divisioncapitalasker.cpp:32
msgid "The capital of %1 is..."
msgstr "Prestonica %1 je..."
"|/|"
"Prestonica $[get-form 'genitive %1] je..."

#: mapasker.cpp:177
msgid "Please click on %1"
msgstr "Kliknite na %1"
"|/|"
"Kliknite na $[get-form 'accusative %1]"

Again, these scripts are intuitive. We must only define function get-form in scripting module, same as set-form from before. So here they are:

(define form-hash (make-hash-table 31))

(define-public (set-form case form . others)
    (hash-set! form-hash (cons case (msgstr)) form)
    (if (not (null? others))
        (apply set-form others)
        (msgstr)))

(define-public (get-form case base)
    (hash-ref form-hash (cons case base)))

Unfortunately, there is some condensed Scheme code involved here, but it's at least short. And you can keep this in your kdelibs.scm, so that you have dynamic form setting available in any KDE application (as kdelibs.po/.scm is always being read).

One special point to note in the code above are calls to function msgstr. This function reports the ordinary translation, which is being used as part of hash key (the other part being case), and as return value of set-forms (so that the big form-setting interpolation evaluates to ordinary translation, as we said it should).

One can even define set-form in such away that no extra quoting is needed to set forms, and that get-form is not needed at all. That would result in total minimum of scripting syntax:

#: mapsdatatranslation.cpp:115
msgid "Greece"
msgstr "Grčka"
"|/|"
"$[set-form genitive Grčke accusative Grčku]"

#: divisioncapitalasker.cpp:32
msgid "The capital of %1 is..."
msgstr "Prestonica %1 je..."
"|/|"
"Prestonica $[genitive %1] je..."

Alas, to make this possible, set-form will have to be a bit longer and a lot uglier (but hey, you'll have it ready to use...)

Frequently it happens that plural form handling, although being able to produce grammatically correct translations, sounds bad in some language. Take a look at this message from kfindpart.po, translated into Serbian language:

#: kfinddlg.cpp:113 kfinddlg.cpp:205
#, c-format
msgid ""
"_n: one file found\n"
"%n files found"
msgstr ""
"%n fajl je pronađen\n"
"%n fajla su pronađena\n"
"%n fajlova je pronađeno"

In English original, for singular case, instead of a bit awkward "1 file found", programmer put nicer alternative "one file found". Serbian translator cannot do the same thing, because singular form is same as plural form of numbers ending with 1 (sans those endind with 11). In his translation all forms must have plural placeholder %n, producing an awkward solution for singular case. Furthermore, it would be nice if there could also be special form for 0, i.e. when no files are found. The script to achieve all this is here:

#: kfinddlg.cpp:113 kfinddlg.cpp:205
#, c-format
msgid ""
"_n: one file found\n"
"%n files found"
msgstr ""
"%n fajl je pronađen\n"
"%n fajla su pronađena\n"
"%n fajlova je pronađeno"
"|/|"
"$[for-n 0 \"Nijedan fajl nije pronađen\")"
"        1 \"Jedan fajl je pronađen\")]"

Script is again just one big interpolation, containing call to function for-n, which is being given the special forms for 0 and 1. So, for-n should return one of those forms if applicable, or one of ordinary forms otherwise. The definition of for-n would go like this:

(define-public (for-n num form . others)
    (if (= num (string->number (arg 0)))
        form
        (if (not (null? others))
            (apply for-n others)
            (msgstr))))

The part (= num (string->number (arg 0))) compares our special case (num) and plural argument ((string->number (arg 0))). Note how we get the plural argument: first, we call (arg 0), which returns the first argument that came with the message, which is the plural argument in case of plural messages; then, since all arguments come as strings, we must convert it to number using Scheme function string->number. Now, if special case matches the plural argument, we return corresponding form. Otherwise, if there are some special forms left ((not (null? others))), we go on to examine them ((apply for-n others)), and if not, we return ordinary translation ((msgstr)).

What happens if original message contains more than one argument requesting plural form? This case is unsolvable by normal plural handling, the only thing one can do is make it the least bad. In kalarm.po you can find such a case (translation into Serbian):

#: messagewin.cpp:560
msgid ""
"_n: in 1 hour %1 minutes' time\n"
"in %n hours %1 minutes' time"
msgstr ""
"u roku od %n sata %1 minute\n"
"u roku od %n sata %1 minuta\n"
"u roku od %n sati %1 minuta"

Translator could handle only plural of hours, but not plural of minutes. One possible scripted solution would be this:

#: messagewin.cpp:560
msgid ""
"_n: in 1 hour %1 minutes' time\n"
"in %n hours %1 minutes' time"
msgstr ""
"u roku od %n sata i %1 minute\n"
"u roku od %n sata i %1 minuta\n"
"u roku od %n sati i %1 minuta"
"|/|"
"u roku od %n $[plural %n sata sata sati] "
"i %1 $[plural %1 minute minuta minuta]"

This one might be a bit harder to parse at first sight. It is actually a single sentence of the form "u roku od %n $[...] i %1 $[...]", where the two interpolations are calling function plural with appropriate argument and three forms, one of which should be returned based on argument value. The first interpolation gets %n and forms of "hours", and the second %1 and the forms of "minutes".

Function plural is to be defined in a scripting module, and it is obviously language dependent. We define it as a macro, so that we can use it as shown above, without need to quote forms which are just single words. For Serbian language, here is how it could look like:

(define-macro (plural numstr form1 form2 form3)
    (cond
        ((string-suffix? "11" numstr) (->string form3))
        ((string-suffix? "12" numstr) (->string form3))
        ((string-suffix? "13" numstr) (->string form3))
        ((string-suffix? "14" numstr) (->string form3))
        ((string-suffix?  "1" numstr) (->string form1))
        ((string-suffix?  "2" numstr) (->string form2))
        ((string-suffix?  "3" numstr) (->string form2))
        ((string-suffix?  "4" numstr) (->string form2))
        (else (->string form3))))

Function cond takes a look at cases of passed number string (numstr) and chooses one of three forms according to Serbian language rules. cond will go through conditions in turn, and return the form associated to first condition that is true; if none is, the form corresponding to final else is returned.

Function string-suffix? is built in and checks if the first argument string is suffix of second argument string, thus checking number endings in this case. Function ->string is a custom one, used here to make forms into strings in case they weren't supplied as such (like in the previous script). You can define it like this:

(define-public (->string thingy)
    (cond
        ((string? thingy) thingy)
        ((symbol? thingy) (symbol->string thingy))
        ((number? thingy) (number->string thingy))
        (else (error "Cannot convert into string:" thingy))))

Frequently it happens that programmer didn't implement plural handling, using instead only plural or ending nouns with "(s)". Although this is usually considered as a bug in application, and therefore should be promptly fixed, KDE applications are full of such cases.

Moreover, it sometimes happens that implementing plural handling is both unnecessary from the point of English language, and would be very tedious (or even impossible) to do. Consider following example from konversation.po, translated into Serbian language:

#: ircinput.cpp:310
msgid ""
"You are attempting to paste a large portion of text "
"(%1 bytes or %2 lines) "
"into the chat. This can cause connection resets or flood "
"kills. Do you really want to continue?"
msgstr ""
"Pokušavate da prenesete veliki deo teksta "
"(%1 bajtova ili %2 linija) "
"u ćaskanje. To može izazvati resetovanja veze ili poplavna "
"izbacivanja. Želite li zaista da nastavite?"

From the context, "a large portion of text", it is obvious that singulars will never occur, so having only plural forms is perfectly fine for English. However, if you recall discussion about plural forms in Serbian in example of additional plural forms, you can see that ordinary translation cannot cover these cases.

But what can programmer do about this problem, especially considering that everything is fine in English, so any further work becomes unnecessary overhead? He would have to do some acrobatics of handling both plurals outside of the message and than splitting them in, risking that it will still not work for some languages.

Instead, it is much easier for translator to script it, using function plural, as described in example of multiple plurals. Here is how that script would look like:

#: ircinput.cpp:310
msgid ""
"You are attempting to paste a large portion of text "
"(%1 bytes or %2 lines) "
"into the chat. This can cause connection resets or flood "
"kills. Do you really want to continue?"
msgstr ""
"Pokušavate da prenesete veliki deo teksta "
"(%1 bajtova ili %2 linija) "
"u ćaskanje. To može izazvati resetovanja veze ili poplavna "
"izbacivanja. Želite li zaista da nastavite?"
"|/|"
"Pokušavate da prenesete veliki deo teksta "
"(%1 $[plural %1 bajt bajta bajtova] ili "
"%2 $[plural %2 linija linije linija]) "
"u ćaskanje. To može izazvati resetovanja veze ili poplavna "
"izbacivanja. Želite li zaista da nastavite?"

Script is the same sentence as ordinary translation, but with fixed translations of "bytes" and "lines" replaced by interpolations of function plural on appropriate arguments and forms.

Simple messages are those one taking no arguments (i.e. no placeholders in it) and requiring no scripting. Since normally most (say 99%) of messages wouldn't be scripted, this case will show the pure overhead caused by the scripting infrastructure.

The i18n call inside the loop references random messages from kdelibs.po. The code looks like this (vector msgids contains UTF8 encoded msgids):

srand(0);
for (uint i = 0; i < BIGNUM; i++)
    i18n(msgids.at(rand() % msgids.size()));

In this example, message is taking three arguments, all of which must have proper grammatical form. This represents an exaggeration of one expected case when scripting will be used, to obtain proper forms of words or phrases in different contexts.

The i18n code is this:

QValueList<QString> ra; // Radio alphabet, 26 words.
ra.append("Alpha"); ra.append("Bravo"); ra.append("Charlie");
...
ra.append("Xray"); ra.append("Yankee"); ra.append("Zulu");

uint len = ra.size();
srand(0);
for (uint i = 0; i < BIGNUM; i++)
    i18n("Just checking: %1, %2, %3...\n")
        .arg(ra[rand() % len])
        .arg(ra[rand() % len])
        .arg(ra[rand() % len]);

And the scripted translation in PO file is:

msgid "Just checking: %1, %2, %3..."
msgstr "Samo proveravam: %1, %2, %3..."
"|/|"
"Samo proveravam: $[radio %1], $[radio %2], $[radio %3]..."

Workhorse of this script is function radio, defined in scripting module. For efficiency of search, it uses Guile's hash tables with symbols as keys:

(define radio-codes (make-hash-table 31))
(hash-set! radio-codes 'alpha "avala")
(hash-set! radio-codes 'bravo "beograd")
(hash-set! radio-codes 'charlie "cetinje")
...
(hash-set! radio-codes 'xray "iks")
(hash-set! radio-codes 'yankee "ipsilon")
(hash-set! radio-codes 'zulu "zagreb")

This means that arguments passed to the script must be normalized, in order to be transformed into symbols for hash keys. Normalizing is done by removing any spaces, dashes and parenthesis from the argument string, and then lowercasing it; the function norms:

(define (norms str)
    (ustring-downcase
        (string-delete str
            (lambda (c) (or
                (char=? c #\space)
                (char=? c #\-)
                (char=? c #\()
                (char=? c #\)))))))

Finally, function radio has this form:

(define-public (radio code)
    (let ((trcode (hash-ref radio-codes (string->symbol (norms code)))))
        (if trcode trcode code)))

In first line it normalizes string code, transforms it to symbol and tries to find its value in the hash. In second line it returns the translation if found, otherwise the original argument.

When message has plural forms, translator may want some forms additional to those represented by KDE rules for his language (eg. for 0 or 1). This is also one expected usage of scripting, contains more processing of arguments than in FORMS case, and stresses the i18n call for scripted plural handling, implementation of which has biggest overhead (compared to ordinary and context-info i18n calls).

The loop with i18n call is simple:

srand(0);
for (uint i = 0; i < BIGNUM; i++)
    i18n("Commissioned one ship in total.",
         "Commissioned %n ships in total.", rand() % 100);

The programmer has used nicer wording for case of unity, putting "one" instead of awkward "1". Translator may not be able to do so, if in his language the form for 1 is needed for other numbers as well. Therefore he writes the special plural handling function, which can deliver forms for 0 and 1 as special cases, and normal forms otherwise. The scripted translation in PO file looks like this:

msgid "_n: Commissioned one ship in total.\n"
"Commissioned %n ships in total."
msgstr ""
"Porinut je ukupno %n brod.\n"
"Porinuta su ukupno %n broda.\n"
"Porinuto je ukupno %n brodova."
"|/|"
"$[plural-zero-one %n "
"    \"Nije porinut nijedan brod.\" "
"    \"Porinut je samo jedan brod.\"]"

Here, function plural-zero-one will take the plural argument and two special forms, for 0 and 1. This function is defined in the scripting module like this:

(define-public (plural-zero-one numstr spec0 spec1)
    (let ((num (string->number numstr)))
        (cond
            ((= num 0) spec0)
            ((= num 1) spec1)
            (else (msgstr)))))

plural-zero-one first converts the argument to Guile number (remember, all arguments to scripts are passed as strings), checks if it is 0 or 1, and returns appropriate special plural form if so. Otherwise, it returns result of msgstr function, which is the non-scripted translation, and is exactly one of the ordinary plural forms that we need in this case.

The performance is measured in thousands of translated messages per second, kmsg/s. This measurement unit gives nice numbers to compare, but to it one can also associate a certain absolute meaning: if a menu with 50 entries mustn't spend more than 50 ms (0.05 seconds) getting its translations, then i18n layer must be able to deliver round performance of 1 kmsg/s.

Aside from pitting the scripted against non-scripted KDE, also of interest is scripted KDE having scripting turned off by the user (so he gets only ordinary translations).

The results on an AthlonXP 2000 (1.67 GHz) machine, using GCC 4.0.1, are shown in the following tables:

On a not too fast machine by today's standards, in the worst case (FORMS) the scripting system was able to deliver 6.77 kmsg/s, much more than what is tentatively needed (1 kmsg/s). Furthermore, results show that when scripting is turned off, user will certainly not be able to feel any difference in performance.

One other thing of possible interest is the distribution of total execution time between the run of Guile interpreter and all i18n code executed arround it. This distribution is given by following table.