Imagine a program which provides a "fuzzy" clock. A fuzzy clock displays the time in prose and rounded, like "four o'clock", "quarter to nine", or "five past midnight". This program has a graphical display on the desktop background or in a desktop panel. The day is divided into several periods, and the current period is shown alongside the time, like "quarter to nine (evening)". Additionally, the period text is shown in a dimmer font, like grayish as opposed to black for time.

We have two implementations of this fuzzy clock. One is written in C++ as the programming language and Qt as the UI toolkit, and the other in Python and GTK+. Translation-related snippets in the C++/Qt code^[2], using current Gettext interface, would look like this:

// C++/Qt

#define gettext(text) QString::fromUtf8(gettext(text))
// ...to make gettext call immediately result in Qt's QString object.

QString getPeriodString (int hour)
{
    if (hour > 10 && hour <= 4)
        return gettext("night");
    else if (hour > 4 && hour <= 9)
        return gettext("morning");
    ...
}

QString getHourString (int hour)
{
    switch (hour) {
    case 1: return gettext("one");
    case 2: return gettext("two");
    ...
    }
}

QString getTimeString (int hour, int minute)
{
    QString hourString = getHourString(minute <= 32 ? hour : hour + 1);
    if (minute <= 2)
        return gettext("%1 o'clock").arg(hourString);
    ...
    else if (minute > 27 && minute <= 32)
        return gettext("half past %1").arg(hourString);
    ...
    else if (minute > 42 && minute <= 47)
        return gettext("quarter to %1").arg(hourString);
    ...
}

QString getDisplayString (int hour, int minute)
{
    QString timeString = getTimeString(hour, minute);
    QString periodString = getPeriodString(hour);
    return gettext("%1 (<font color='gray'>%2</font>)")
                  .arg(timeString).arg(periodString);
}

The string which puts together the time and period substrings has to be wrapped for translation too, because some languages may needed different spacing and ordering of parts. In Python/GTK+, the implementation would be:

# Python/GTK+

def get_period_string (hour):

    if hour > 10 and hour <= 4:
        return gettext("night")
    elif hour > 4 and hour <= 9:
        return gettext("morning")
    ...


def get_hour_string (hour):

    if hour == 1:
        return gettext("one")
    elif hour == 2:
        return gettext("two")
    ...


def get_time_string (hour, minute):

    hour_string = get_hour_string(hour if minute <= 32 else hour + 1)
    if minute <= 2:
        return gettext("%s o'clock") % hour_string
    ...
    elif minute > 27 and minute <= 32:
        return gettext("half past %s") % hour_string
    ...
    elif minute > 42 and minute <= 47:
        return gettext("quarter to %s") % hour_string
    ...


def get_display_string (hour, minute):

    time_string = get_time_string(hour, minute)
    period_string = get_period_string(hour)
    return gettext("%(a1)s (<span foreground='gray'>%(a2)s</span>)") \
                  % dict(a1=time_string, a2=period_string)

The last string uses named formatting directives (%(a1)s, %(a2)s) to enable the translator to reorder them if necessary^[3]; in the C++/Qt version above, this was implicitly possible.

Let us now extract a few exemplary translation-related lines from the two versions:

// C++/Qt
gettext("one")
gettext("half past %1").arg(hourString)
gettext("%1 (<font color='gray'>%2</font>)")
       .arg(timeString).arg(periodString)

and:

# Python/GTK+
gettext("one")
gettext("half past %s") % hour_string
gettext("%(a1)s (<span foreground='gray'>%(a2)s<s/pan>)") \
       % dict(a1=time_string, a2=period_string)

The first observation to make is that, from the translator's viewpoint, there is no real difference between these two sets of strings, as the user-visible output will be the same. But the strings are technically different because the underlying coding framework shows through them. This difference is mere hindrance, as it adds nothing to the efficiency or quality of translation. In fact, it detracts: sometimes in translation the technical parts of the message (as opposed to user-visible text) should be changed as well, and profusion of formats will cause the translator to hesitate or err.

The second observation is that this code is, in fact, non-translatable. What is "half past four" in English, will be the equivalent of "half to five" in many other languages; whereas it is always "number o'clock" in English, in many languages the equivalent of "o'clock" will have to change according to the number; and so on.^[4] This forces the translator into the notorious "least bad" translation.

The third observation is that if the programmer changes something in the visual aspects of the markup, for example the display color for a certain type of entity (such as for the day period above), all previously translated messages become fuzzy in PO files. This not only leads to unnecessary effort for all translators, but a translator may also fail to spot the change^[5] and unfuzzy the message without properly adjusting the translation. This problem also applies to extended formatting directives such as %.1f or {num:05d}.

Now we convert the fuzzy clock code to use generalized Gettext calls, i.e. to make use of Gettext extensions which will be proposed.

The basic generalized call is named xgettext, where x* stands for "XML-like". Here is how it would be employed in the new C++/Qt version:

// C++/Qt
xgettext("one")
xgettext("half past {hour}").subs("hour", hourString)
xgettext("{time} (<dimmed>{period}</dimmed>)")
        .subs("time", timeString).subs("period", periodString)

and in the new Python/GTK+ version:

# Python/GTK+
xgettext("one")
xgettext("half past {hour}", hour=hour_string)
xgettext("{time} (<dimmed>{period}</dimmed>)",
         time=time_string, period=period_string)

For translators, there is no longer any difference between the two sets of strings. Coding framework specific formatting directives have been replaced with uniform placeholders. Visual markup has been replaced with customizable markup, which enables the programmer to change the resulting visual appearance without perturbing the string seen by the translator. The xgettext function does not return a simple string, but a special object which captures the arguments and substitutes them internally after the translation has been fetched from the catalog. This enables translators to script the translation, such that it depends on the arguments supplied at run time.

In the PO file of the C++/Qt version, the translation looks like this:

#: fuzzyclock.cpp:87
#, ggx-format
msgid "morning"
msgstr "jutro"

...

#: fuzzyclock.cpp:123
#, ggx-format
msgid "one"
msgstr "jedan"
msgscr "$[set-property next-hour dva]"

#: fuzzyclock.cpp:124
#, ggx-format
msgid "two"
msgstr "dva"
msgscr "$[set-property next-hour tri]"

...

#: fuzzyclock.cpp:256
#, ggx-format
msgid "half past {hour}"
msgstr "{hour} i trideset"
msgscr "pola $[get-property next-hour {hour}]"

...

#: fuzzyclock.cpp:312
#, ggx-format
msgid "{time} (<dimmed>{period}</dimmed>)"
msgstr "{time} (<dimmed>{period}</dimmed>)"

The PO file of the Python/GTK+ version would look exactly the same, only the source references would show fuzzyclock.py:... instead. Hence, the translator does not feel the underlying coding framework at all. Messages have a new format flag^[6], ggx-format, which shows that these are generalized Gettext strings with XML-like markup instead of coding framework specific.

What did the translator do here? The first message is quite ordinary. The second message, which starts the hours list, has a translation script in addition to the plain translation. The translator added the script by adding the new msgscr field with the script. Without going into syntax, this script sets the next-hour property with the value "dva" ("two") on the string "jedan" ("one") given by msgstr; the plain meaning being "the next hour after one is two". The hour "dva" ("two") has next-hour set to "tri" ("three"), and so on. Then come the time composition messages. The composition "half past hour" is translated in msgstr as "hour i trideset" ("hour and thirty"), but the user would rather expect the formulation "pola hour+1" ("half hour+1"). The translator now adds a script which overrides the msgstr translation, unless it fails for some reason. This script fetches the value of the next-hour property of the {hour} argument once it is substituted, where the property was set through the previously fetched corresponding hour message, and inserts the value into the surrounding text. The result is the desired "pola hour+1" translation.

The rest of this document describes in detail the features demonstrated in this example, hopefully covering many of the questions that may have been raised.

For the moment putting aside the verbosity due to keyword-like argument placeholders (the section about placeholders will discuss that), the only real inconvenience in the proposed generalized Gettext interface is the need to explicitly call a conversion method to get the resolved (translated and arguments substituted) native string. There are several possibilities to get around this.

The ideal world variant is that different destinations (GUI widgets, shell writers...) are aware of Gettext::Composer; that is, that Gettext::Composer can be an argument wherever the native string can be. This would be that more important when customizable markup is taken into account, because, in principle, it could resolve into different destination markups. For example, a "<emph>...</emph>" segment would resolve into *...* for plain text destinations, and into "<i>...</i>" for HTML-like destinations (e.g. Qt Rich Text or GTK+ Pango Markup tooltips and whatsthis texts). This is presented in much more detail in the section on markup. Since the destination knows which type of markup it can process, it would internally call the appropriate conversion method of Gettext::Composer.

In absence of aware destinations, the second thing that comes to mind is implicit conversion, through e.g. conversion operators in C++ or duck typing in Python. Keeping in mind the previous paragraph, you may observe that one problem with implicit conversion is how to specify the target markup; but this will be discussed in the section on markup. A more pressing problem are inadvertent conversions. Consider this:

// C++/Qt
QStringList quips;
...
quips.append(xgettext("It's full of <emph>what</emph>?!"));
    // ...implicit conversion kicks in, leading to early resolution;
    // <emph> has to be conservatively converted to plain text.
...
QString note = xgettext("A new quip has just been uttered: {quip}")
                       .subs("quip", quips[quipIndex])
                       .toRichString();
    // ...the final message should be rich text, but all substitutes
    // already got converted to plain text earlier.

There can be several layers of code between the composite message and the translated substitutions, so if the problem is detected long after the code has been written, it may be difficult to refactor it.

There can also be a middle ground. It follows from two observations: most of the time deferred argument substitution is not necessary (e.g. a text label is directly written into or precedes a widget constructor call), and most of the arguments are either not translations themselves or are short texts without need for markup (e.g. names of things). There could be a parallel set of immediate-resolution calls, which would take all the arguments at once, call a conversion method internally, and return a native string. Both of the following two lines would set msg to a Python string object:

# Python
msg1 = xgettext("Processing {file}...", file=a_path).to_string()
msg2 = ixgettext("Processing {file}...", file=a_path)

The i* in ixgettext stands for "immediate".

In current C++, an ixgettext call that could both work with keyword placeholders and require no conversion call would be longer than xgettext itself, defeating the purpose. However, a stripped-down variant is still possible, by limiting oneself to ordinal placeholders:

// C++/Qt
QString msg1 = xgettext("Processing {1}...").subs(aPath).toString();
QString msg2 = ixgettext("Processing {1}...", aPath);

Ordinal placeholders index rather than name arguments, and can be used alongside keyword placeholders; the section on placeholders provides more details. ixgettext can now be implemented as a set of templates^[9] like this:

// C++/Qt

inline QString ixgettext (const char *text)
{
    return xgettext(text).toString();
}

template <typename A1>
inline QString ixgettext (const char *text, const A1 &a1)
{
    return xgettext(text).subs(a1).toString();
}

template <typename A1, typename A2>
inline QString ixgettext (const char *text, const A1 &a1, const A1 &a2)
{
    return xgettext(text).subs(a1).subs(a2).toString();
}

...

This puts an arbitrary limit on number of arguments, but 9 is a nice number: it includes all single-digit placeholders, and the number of messages with more than 9 arguments is insignificant^[10] in practice. Also, the C++11 standard introduces variadic templates, which removes the limit as well as the verbosity of definition.

It depends on programmer's sensibility which of these three variants -- explicit conversion, implicit conversion, immediate-resolution -- is preferable. Some programmers may not mind using verbose explicit conversion everywhere, to keep things uniform and having least surprise. Others may opt for the combination of immediate-resolution calls most of the time and explicit conversion when necessary. Yet some will have no grudge against implicit conversion (especially when they do not use any markup in text). Therefore the generalized Gettext library may very well provide all three variants (with implicit conversion that can be disabled), subject to applicability in different programming languages.

In current Gettext, the translation call for plural forms ngettext takes an explicit integer argument to decide the plural form. Normally this number is also shown in the string, so it needs to be repeated twice in the logical composition of the call:

# Python
msg = ngettext("%d file selected", "%d files selected", nselfiles) \
              % nselfiles

This is necessary because ordinary Gettext calls have no knowledge of arguments which will later be substituted. With argument capturing, however, this is no longer the case, and the generalized plural call nxgettext can look like this:

# Python
msg = nxgettext("{numsel} file selected", "{numsel} files selected",
                numsel=nselfiles)

At first sight, this may look like a small syntactic gain for no real benefit. And it may even look like a trouble ahead: what happens when the message has several arguments and more than one of them is an integer? Like in this example:

# Python
msg = ngettext("%(numsel)d of %(numtot)d file selected"
               "%(numsel)d of %(numtot)d files selected", nselfiles) \
              % dict(numsel=nselfiles, numtot=ntotfiles)

In fact, this is where the generalized approach shines. The programmer can always explicitly set which argument decides the plural by adding the !n extension to the appropriate placeholder:

# Python
msg = nxgettext("{numsel} of {numtot!n} file selected",
                "{numsel} of {numtot!n} files selected",
                numsel=nselfiles, numtot=ntotfiles)

This means that the translator can change which argument decides the plural, and that is exactly what could be needed for some languages, as in this example:

#: file_selector.py:429
#, ggx-format
msgid "{numsel} of {numtot!n} file selected"
msgid_plural "{numsel} of {numtot!n} files selected"
msgstr[0] "Izabrana {numsel!n} datoteka od {numtot}"
msgstr[1] "Izabrane {numsel!n} datoteke od {numtot}"
msgstr[2] "Izabrano {numsel!n} datoteka od {numtot}"

Precise rules for plural resolution based on the placeholders and argument types are given in the section on placeholders.

Another benefit derives from lazy translation resolution, which means that deferred plural resolution is possible:

# Python

def process_files (...):
    ...
    msgbase = nxgettext("Processing, {num} files remaining...")
    prgup = ProgressUpdater(..., msgbase)
    ...

class ProgressUpdater:

    def __init__ (..., msgbase):
        """
        ...
        @param msgbase: ... must be plural with {num} placeholder...
        ...
        """
        self._msgbase = msgbase
        ...

    def update (self):
        ...
        msg = self._msgbase.subs("num", len(remaining_items)).to_string()
        ...

In practice, in some plural messages the singular string does not contain the plural-deciding argument, for stylistic reasons. This would still be supported:

# Python
msg = nxgettext("Move this file to {dest}?",
                "Move these {num} files to {dest}?",
                num=nfiles, dest=dirpath)

The difference is that now the translation call knows which placeholder is linked to the argument that decides the plural, so it can complain at run time (e.g. in debug mode) if there is any other difference in placeholder sets between the singular and the plural string.^[11]

The disambiguation context call pgettext remains the same in the generalized variant:

// C++
pxgettext("default action", "Default")
pxgettext("default shortcut", "Default")

The context string, however, may serve some additional purposes to its basic disambiguation purpose, as will be described in the section on interface contexts in relation to customized markup.

The explicit domain selection call dgettext, which is for example necessary in libraries, also remains the same:

// C++
dxgettext("pycardlib", "Available decks:")

In practice, explicit domain selection calls are wrapped in a macro definition, in order not to have to repeat the domain name argument in every call:

// C++
#define _(msg) dxgettext("pycardlib", msg) // in a header file
....
_("Available decks:") // in a source file

This still works with generalized calls, but since generalized calls will have specialized bindings for each coding framework, it would be nice to always define a domain selection mechanism which feels "the most native" to that coding framework.

In generalized plural calls, there is no explicit plural-deciding number given as the call argument. Instead, the translation object (returned by the generalized translation call) selects one of the arguments as plural-deciding, at the moment when it gets converted into a native string (after all arguments have been supplied). How the plural-deciding argument gets selected was partly covered in the section on argument capturing, what remains to do here is to define how this works with different types of placeholders.

When only one argument is substituted, then obviously that argument is the plural-deciding one independently of the placeholder type:

# Python
nxgettext("{numsel} file selected", "{numsel} files selected",
          numsel=nselfiles)
nxgettext("{1} file selected", "{1} files selected", nselfiles)
nxgettext("{} file selected", "{} files selected", nselfiles)

When more than one argument is substituted, then it depends on placeholder types. If there are any ordinal placeholders in the string, an integer type argument which corresponds to the lowest ordinal placeholder is selected. Here the ntotfiles argument is deciding plural:

# Python
nxgettext("{2} of {1} file selected", "{2} of {1} files selected",
          ntotfiles, nselfiles)

If there are empty placeholders in the string, then after they get assigned ordinals, the behavior is the same as with ordinal placeholders. If there are no ordinal or empty placeholders which correspond to an integer argument, but there is exactly one keyword placeholder which does correspond to an integer argument, that argument is used to decide plural:

# Python
nxgettext("Move {num} files to {destdir}?",
          "Move {num} files to {destdir}?",
          num=num_files, destdir=dest_dirpath)

In any other case, the plural-deciding argument must be explicitly marked using the !n placeholder extension:

# Python
msg = nxgettext("{numsel} of {numtot!n} file selected",
                "{numsel} of {numtot!n} files selected",
                numsel=nselfiles, numtot=ntotfiles)

These rules mean that, in practice, plural selection "just works" as long as there is exactly one number argument in the message; and as soon as there are two numbers, even if implicit selection would do, it is best to explicitly mark the plural-deciding argument.

The translator can always use explicit marking, whether it was used in the original string or not. This means that if there are two or more numbers in the message, the translator can also change which argument decides the plural (cf. the example in argument capturing section).

If the plural-deciding argument cannot be automatically selected, or the explicitly marked argument is not of an integer type, an arbitrary number (at best random) is used to decide the plural form and (possibly in debug mode only) a problem is signaled.

Up until now, the terms "formatting directive" and "placeholder" were used more or less interchangeably, but let us now define a concrete difference between them. A placeholder only indicates the position where the argument is substituted, and which argument is substituted. A formatting directive additionally provides type conversion and formatting sequences for certain types of arguments, such as %+12.4f or ~12,4@f for floating point numbers.

Should then there really be only uniform placeholders, or actually uniform formatting directives? The answer is definitely -- only placeholders. There should be no argument formatting sequences within translatable strings. There are several reasons for this strong position.

The first reason is based on a survey of existing translations. The Gnome Translation Project contains a large number of user interface catalogs with printf style formatting directives, to a greater or lesser extent translated into over 160 languages. It is therefore a good single place to examine how frequently translators modified formatting directives^[14] as found in the original text, and what were the changes about. The examination of Gnome 3.0 (Gnome core modules) translations yielded the following results:

These findings make it clear that formatting sequences are extremely rarely modified by translators, and when they are, almost exclusively unintentionally. Furthermore, there are many more modifications which result in lost information or reduced comprehensibility alone than the intentional modifications. It is much better to handle language-specific argument formatting by relying on the locale library of the coding framework (which provides all of the number, time, date, etc. formats in various verbosity levels), or, in the rare cases when that is not sufficient, by using a well-commented meta-message which asks translators only for necessary formatting details.

The second reason why formatting sequences in translatable strings are a bad idea was demonstrated by the same survey. A set of around 60 modified formatting directives (out of 151 total) went like this, in various languages:

#. TRANSLATORS: tell the user how much time they have got
#: ../src/gpm-manager.c:1505
#, c-format
msgid "%s of battery power remaining (%.0f%%)"
msgstr "Απομένουν %s λειτουργίας της μπαταρίας (%.1f%%)"

#. TRANSLATORS: tell user more details
#: ../src/gpm-manager.c:1637
#, c-format
msgid "Wireless mouse is low in power (%.0f%%)"
msgstr "Bezvadu peles baterija ir gandrīz tukša (%.1f%%)"

#. TRANSLATORS: tell user more details
#: ../src/gpm-manager.c:1644
#, c-format
msgid "Wireless keyboard is low in power (%.0f%%)"
msgstr "Haririk gabeko teklatuak energia baxua du (%% %.1f)"

What had happened here is that the programmer decided, between Gnome 2.32 and Gnome 3.0, that displaying charge percentages for various devices using one decimal is needlessly verbose, so he switched to rounding to nearest integer. He went through all the affected messages in the code changing %.1f into %.0f, making them fuzzy in all translations. Some translators didn't notice the change when unfuzzing those messages, and hence didn't propagate the readability improvement of the original text into the translation. Had the code been written to have messages like this:

#. TRANSLATORS: tell user more details
#: ../src/gpm-manager.c:1644
#, ggx-format
msgid "Wireless keyboard is low in power ({charge}%)"
msgstr ""

it would have been possible to change the formatting in a single place (more on this shortly), and there would have been no extra work at all for translators.

The third and final reason to keep formatting sequences out of messages has to do with this question: what kind of formatting sequences would be available? If the choice would be, say, a subset of printf formatting sequences, many programmers in non-C coding frameworks would feel overly constrained. For example, in Python's string.format formatting sequences are completely free, in that each class can define a method to parse its own formatting directive syntax. A similar question is that of which locale provider to use for formatting. For example, a GTK+-based program on a GNU system would use glibc functionality and locale settings, but a Qt-based program would use Qt's locale functionality and settings. These questions are impossible to answer to everyone's satisfaction, and would both complicate the implementation and impede the acceptance of generalized Gettext.

Instead of in-string formatting sequences, argument formatting would be delegated to different bindings of generalized Gettext. In Python, for example, .subs methods would take the formatting sequence as the third argument:

# Python
charge_fmt = ".1f"
...
if device == DeviceType.Mouse:
    msg = xgettext("Wireless mouse is low in power ({charge}%)")
elif device == DeviceType.Keyboard:
    msg = xgettext("Wireless keyboard is low in power ({charge}%)")
...
msg = msg.subs("charge", device_charge, charge_fmt)

Variadic-keyword calls could treat two-tuples as pairs of argument and its formatting sequence:

# Python
xgettext("Wireless mouse is low in power ({charge}%)",
         charge=(mouse_charge, charge_fmt))

In C++/Qt, .subs methods would take over the argument-based formatting of QString::arg methods:

// C++/Qt
xgettext("Wireless mouse is low in power ({charge}%)")
         .subs("charge", mouseCharge, 0, 'f', 1)

.subs methods would be implemented as thin wrappers around coding framework's native argument formatting facilities. In the Python example above they would internally execute ("{:%s}" % fmtseq).format(arg) to get the formatted argument, whereas in the C++/Qt example they would execute QString("%1L").(arg, fieldWidth, format, precision). In this way, coding framework's locale functionality would be automatically applied as well.

Having defended the position that formatting should be kept out of messages, corner cases may turn up where in-string formatting is the least bad of all available options. For this reason, in-string formatting could actually be allowed, if strongly advised against in the Gettext documentation (including the rationale), and made to require explicit activation, per PO domain. Activation could go something like this:

// C++
bindtextdomain("fooapp", LOCALE_DIR);
textdomain("fooapp");
Gettext::Composer::setFormatting("fooapp", true);

After that, formatting sequences could be specified as :fmtseq placeholder extensions (like in Python's string.format):

// C++
xgettext("Wireless mouse is low in power ({charge:.1f}%)"))
        .subs("charge", mouseCharge)

If the formatting extension is used on a placeholder but in-string formatting is not activated for the PO domain to which the call belongs, the formatting sequence would simply be ignored. Formatting sequences could contain any characters whatsoever (with braces escaped by doubling), and would be terminated by the placeholder's closing brace. This means that the formatting extension must be the last extension in a placeholder.

Another important thing in presence of in-string formatting is how message extraction and translation validation would behave. Since formatting sequences can be arbitrary, the only possibility would be for xgettext to simply accept everything between the colon and the closing brace as a valid formatting sequence, and for msgfmt to perform no checks on it. This may seem like an uncomfortable gap in coverage, but pure placeholder correspondence (keywords, ordinals) would still be checked, and, as we have seen before, formatting sequences can be logically wrong even when syntactically correct.

If it were a usual static markup language, the hardest part about defining generalized Gettext markup would be to come up with the appropriate set of tags and attributes. Even if the set of tags would be extensive, someone would still miss something; worse yet, people would feel bad when forced to pick one of the several tags none of which is quite what they want.^[16] Some people would feel that tag names such as <filename> are too verbose, and would rather have such a frequently used tag be called just <file>, or even <fn>. Some would consider semantic markup an unnecessary complication and be satisfied with visual markup (especially when there is a single known output destination).

A related problem is that of what resolution pattern should be used for a particular tag, when converted into a given target markup. In a HTML-like target markup, some would want <filename>...</filename> to become <b>...</b>, while others would want <tt>...</tt>. Taking one step back, what target markups would be available at all? What about problem-specific, ad-hoc markups?

The proposition to handle these issues is twofold.

Firstly, generalized Gettext markup would provide a built-in extensive set of tags, even synonymous, such as all of <filename>, <file>, and <fn>. It would provide built-in resolution patterns for those tags for a number of widely used user interface markups, such as Qt Rich Text and Pango Text Attributes. New tags and new target markups would be periodically added, as people contribute them. Criteria for accepting tags would not be too strict, e.g. adding a <player> tag would be fine, while an <albatross> tag probably would not.

Secondly, there should be a facility for programmers to define custom tags and target markups in their code, as well as to override resolution patterns for known target markups. The programmer would first search the list of built-in tags for an appropriate tag, trying out a few synonyms; if there is none, the programmer would simply define a new tag. Or, built-in tags could be ignored altogether. Custom tags would be grouped under a name same as the name of a chosen PO domain. Custom resolution patterns would be given per tag (and possibly attribute combinations) and arbitrary target markup name.

Assuming that one does not want to rely on any of the predefined tags and target markups, here is how a complete definition of a small set of custom tags and resolution patterns would look like^[17]:

# Python/GTK+

def setup_markup ():

    domain = "fooapp"

    from xgettext import markup_def
    markup_def(domain, "plain", "place", [], "'{_text_}'")
    markup_def(domain, "pango", "place", [], "<i>{_text_}</i>")
    markup_def(domain, "plain", "unit", [], "'{_text_}'")
    markup_def(domain, "pango", "unit", [], "<u>{_text_}</u>")
    markup_def(domain, "pango", "unit", ["color"],
        "<u><span foreground='{color}'>{_text_}</font></u>")

    from xgettext import markup_set_escape, escape_xml
    markup_set_escape(domain, "pango", escape_xml)

    from xgettext import markup_set_class_plain, markup_set_class_rich
    markup_set_class_plain(domain, "plain")
    markup_set_class_rich(domain, "pango")


def report_event (...):
    ...
    elif ...:
        msg = xgettext("<unit color='{co}'>{un}</unit> enters "
                       "<place>{pl}</place>.",
                       co=unit1.owner().color(), un=unit1.name(),
                       pl=place.name())
    ...
    write_to_event_view(msg.to_rich())
    write_to_log(msg.to_plain())

markup_def(domain, markup, tag, attributes, pattern) call defines a valid combination of a tag and its attributes, and its resolution pattern, for the given domain and target markup. In this example, <place>-tagged segment resolves into single-quoted segment in plain text, and into italic segment in rich text provided by Pango markup. <unit>-tagged segment resolves also into single-quoted segment in plain text, but into underlined segment in rich text. If there is also the color= attribute to <unit>, then the resulting segment in rich text is additionally colored. But, there is no definition of <unit> with color= for plain text -- this means that resolution to plain text falls back to attributeless <unit> definition, effectively ignoring the color= attribute. Resolution patterns use the familiar braced placeholder syntax for substituting element text and attribute values, where {_text_} automatically denotes the element text. The substitutes may need to be escaped in the target markup, so an escaping function can be set using markup_set_escape(domain, markup, escapefunc) call. After custom markup has been defined, markup_set_class(domain, markup) are used to link markup classes to target markups, so that to_class methods of translation objects produce expected native strings.

Simple replacement resolution pattern may not always be sufficient. It would therefore be possible to use an arbitrary function instead. This function would get as arguments the complete processing context: the current tag, its attributes, the element text, and the path of parent tags from top. It would be even possible to have a single markup_def call per target markup:

# Python/GTK+

def setup_markup ():

    ...
    markup_def(domain, "plain", resolve_to_plain)
    markup_def(domain, "pango", resolve_to_pango)
    ...


def resolve_to_pango (tag, attrs, text, path):

    if tag == "place":
        return "<i>%s</i>" % text
    elif tag == "unit":
        if "color" in attrs:
            return ("<u><span foreground='%s'>%s</font></u>"
                    % (attrs["color"], text))
        else:
            return "<u>%s</u>" % text
    return text

Availability of the parent path enables to contextually resolve the tag. A typical example of when this is needed is an emphasis within an emphasis.

Resolution patterns may contain parts which are language dependent. In the previous example, these are (at least) the patterns containing single quotes, since quoting standards vary across languages. This is handled simply by exposing the resolution pattern itself for translation, albeit with a small twist:

markup_def(domain, "plain", "place", [],
           ptgettext("a place name in plain text",
                     "'{_text_}'").to_string())

Since this message is a meta-message with respect to markup, anything that looks like markup in it must be ignored on resolution to native string. This is why the ptgettext call is used, which is the context variant of tgettext basic call. *tgettext calls are also generalized Gettext calls, where t* stands for "plain text": unlike *xgettext calls, *tgettext calls do not process markup. In order for the translation of this message to be properly validated (see the section on XML vs. plain text calls), the message will get the ggt-format flag and not the ggx-format.

Instead of defining a custom markup from the ground up, it would be possible to import markup defined by another PO domain and then add to it and adapt it. This implies that the code which uses that other domain has been already loaded and executed its markup setup, i.e. an underlying library. This is how the built-in markup of the generalized Gettext would be used in the first place:

# Python/GTK+

def setup_markup ():

    domain = "fooapp"

    from xgettext import markup_import, markup_def
    # ...the Gettext library is loaded and built-in markup created.

    # Declare built-in markup to be in use in this domain.
    # "gengettext" is the PO domain of generalized Gettext library.
    markup_import(domain, "gengettext")

    # Define additions and overrides.
    markup_def(domain, ...)
    markup_def(domain, ...)
    ...

A usual element of XML-based markups are custom entities. The default set of entities, needed for escaping, is &, <, >, ", ', but the author can define arbitrary custom entities. It would not be hard to support this in generalized Gettext markup:

# Python/GTK+

def setup_markup ():
    ...
    markup_set_entity(domain, "thisprog",
                      xgettext("FooProcessor").to_string())
    ...

def some_func ():
    ...
    msg = xgettext("If you enable frobaz, &thisprog; will [...]")
    ...

This example looks attractive. If &thisprog; entity is used in place of the program name everywhere, then it becomes "easy" to later change the name, or change the exact spelling, by changing a single line of the code. The program name is exposed to translators for adaptation (translation, transliteration); if the name changes, only this message becomes fuzzy in the PO file, and translators can adapt it again.

Unfortunately, there is a glaring problem with this scheme. In many languages, nouns change their form depending on the function in the sentence. For example, the name "Okteta" has case declensions "Oktete", "Okteti", "Oktetu", "Oktetom" in my native language. Then, other parts of the sentence may depend on properties of the inserted name. For example, "Program started" would be "Okular pokrenut" but "Okteta pokrenuta", because the grammatical gender of "Okular" is masculine, and of "Okteta" feminine.

This means that custom entities are too dangerous to be allowed, even if in some particular cases they could be properly used. When such a case really happens, it is possible to use ordinary argument substitution instead.

Generalized Gettext can, however, provide a fixed set of entities beyond the default XML set. They would be added when judged generally useful and not problematic for translations. An example good candidates are entities for various whitespace characters, such as non-breaking space ( ), which are hard to spot and may be unintentionally replaced if written out literally.

In current Gettext, there is no formalized way to validate coding framework specific markup. msgfmt with --check option will simply treat anything that is not formatting directive as literal text. Invalid markup in translation is normally not as big an issue as invalid formatting directives, since markup processors will try to do their best to render the text or simply strip invalid markup. But it is an issue nonetheless, and it would be a pity not to do anything about it when having Gettext's own markup.

Markup validation splits into two sources and two forms. The two sources are translation and the original text. Either can have invalid markup, and there should be a way to validate both. The two forms are validation at run time and static validation. Markup can obviously be checked at the moment of resolution, at run time (when translation's object conversion method gets called), and warnings can be produced if it is not valid. But run time validation may be too late, especially for translations which get less testing during development than the original text, so static validation would be nice to have.

What are the options for static markup validation? For XML-based fixed formats, the simplest widely used validation option is a DTD (document type definition). But it has two problems. The first is that, since here we talk about customizable markup, it would mean that everyone customizing something would need to write an accompanying DTD file. The second problem is that the complete markup text may not exist statically at all, since it is frequently composed at run time from separate pieces (e.g. paragraphs in a help tooltip). This precludes application of a DTD, or any other standard XML validation scheme.

Luckily, there are two exploitable specificities of user interface text markup when compared to typical text document markup.

The first specificity is that the biggest "document" in user interface is a title and a few paragraphs, possibly with a table or a figure. If one such piece of markup text is invalid, it will not interfere with all the other text in the user interface. And when it is invalid, the markup transformation engine can be forgiving and work around problems. There is no need to simply display an error and leave the user without any text at all. (This is also how current coding framework specific lightweight markups work.)

The second specificity is related to translation. An examination of markup-related errors (that could be detected) in KDE Translation Project shows that over 95% of errors were about lack of well-formedness. Such errors are usually introduced by omissions, typos, and reformulations of the text; also on unfuzzying, when markup in original text was corrected. For example, in the following message the closing </a> was omitted:

msgid ""
"<b>Note: [...] See <a href='[...]'>LensFun project web site</a> "
"for more information.</b>"
msgstr ""
"<b>Hinweis: [...] Für mehr Informationen besuchen Sie "
"<a href=\"[...]\">die Internetseite des LensFun-Projekts</b>."

and in this one the first opening <b> tag name was mistyped as <bb>:

msgid ""
"[...] For example, the <b>uncrustify</b>, <b>astyle</b> or "
"<b>indent</b> formatters can be [...]"
msgstr ""
"[...] Par exemple, les outils de formatage <bb>uncrustify</b>, "
"<b>astyle</b> ou <b>indent</b> peuvent [...]"

In the rest 5% of detected cases, an unknown tag or an unknown attribute was used while markup remained well-formed. Although no other types of cases could be detected by the used validation tool (such as invalid tag nesting), experience puts their upper limit to trace amounts.

These two points lead to a lightweight possibility for "almost validating" markup.

First, it would be required that every message is well-formed on its own, so that well-formedness can be statically checked. This means that the following splitting would result in two invalid messages, although the final composition would be valid:

from gengettext import xgettext, Composer
...
start = xgettext("<para>If you click [...] get the desired effect. ")
if something == "foo":
    insert = xgettext("This will work only [...] is enabled. ")
else:
    insert = ""
end = xgettext("After that, click [...].</para>")
msg = Composer("{}{}{}", start, insert, end).to_string()

The splitting should be instead done as^[18]

para = xgettext("<para>If you click [...] get the desired effect. "
                "{optional-sentence} After that, click [...].</para>")
if something == "foo":
    insert = xgettext("This will work only [...] is enabled.")
else:
    insert = ""
msg = para.subs("optional-sentence", insert).to_string()

A non-well formed message would have either its tags stripped (in release mode) or replaced with conspicuous error indicators (in debug mode), so that the programmer is nudged to fix it.

Then, if the original text in messages is always well-formed, the same can be required of the translation. That alone would catch over 95% of all markup-related translation errors made in practice. Taking into account that markup transformation engine is forgiving, it may even be fine to stop there and declare checking well-formedness to be sufficient validation.

To additionally catch unknown tags and attributes in translation is already much harder, at least organizationally. This is because the validation tool would have to know which tags and attributes are usable in the given PO domain. One way to provide this information -- perhaps too crude -- could be to list all tags and attributes in a simple format, in a special comment that xgettext would extract into POT file header:

def setup_markup ():

    ...

    # Functional markup definition...
    markup_import(domain, "gengettext")
    markup_def(domain, "pango", "action", [], ...)
    markup_def(domain, "pango", "place", [], ...)
    markup_def(domain, "pango", "place", ["link"], ...)
    markup_def(domain, "pango", "unit", [], ...)
    markup_def(domain, "pango", "unit", ["link"], ...)
    markup_def(domain, "pango", "unit", ["color"], ...)
    markup_def(domain, "pango", "unit", ["link", "color"], ...)

    # ...and specification for extraction.
    # xgettext: markup-spec type1
    #   >xgettext;
    #   action:;
    #   place: link;
    #   unit: link color;

The # xgettext: markup-spec type1 tells to xgettext that the rest of the comment is a "type-1" markup specification; this leaves open the possibility for more sophisticated markup specifications in the future. Type-1 specification is a list of semicolon-terminated entries, where each entry may be an inclusion of markup from another PO domain (>domain;) or a tag and its attributes (tag: attr1 attr2 ...;). This would appear in the POT header as:

msgid ""
msgstr ""
"Project-Id-Version: fooapp 1.3.5\n"
"..."
"Customizable-Markup: type1; >xgettext; action:; place: link; "
"unit: link color;\n"

When the validator sees a domain inclusion in the specification, it would in turn parse the header of that other domain for its markup specification. This leads to a problem of how to locate the appropriate PO/POT file for that domain.

The following question may spring to mind, especially if the original text is always well-formed: instead of going to lengths specifying markup, why not simply require that the translation contains the same tagging as the original? The translator would be able to change relative positions of elements, but in the end the translation would have to contain exactly the same elements, and to nest them in the same way, as in the original text. The problem is that this would be overly constraining:

In user interfaces it is frequently possible to activate a visible item, such as a button or a menu entry, by pressing a modifier key and a letter assigned to that item, instead of clicking on it. (This is not the same concept as command shortcuts, where pressing one or several modifier keys and a character key will activate an action regardless of the UI element that can be clicked on for the same purpose.) This activation letter, called the accelerator, is most usually assigned to the item by placing a special character in item's text label.^[19] In this example:

msgid "&Roll Dice"
msgstr "&Baci kockice"

the accelerator in the original text is the letter R, and in the translation the letter B. The character used to mark the accelerator, here ampersand (&), is called the accelerator marker.

There are two problems with accelerator markers when it comes to translation processing (there are more problems in general, but these are beyond the scope of this document). The first problem is that accelerator markers differ between coding frameworks: in Qt it is ampersand (&), in GTK+ underscore (_), in OpenOffice tilde (~), etc. This means that, for example, a search tool cannot know which character to ignore, and a spell-checking tool cannot with certainty eliminate the accelerator marker from the middle of a word. The second problem is that it is not known if the text can contain the accelerator marker at all, to escape it (and how) if a literal character is needed in translation.

For these reasons, generalized Gettext must also introduce uniform accelerator markers. This includes selecting the accelerator marker character, the way to escape it, and making this work in every message. This discussion naturally falls within the area of markup, because accelerator marking is nothing but markup in disguise, a shorthand for something like this:

msgid "<acc>R</acc>oll Dice"
msgstr "<acc>B</acc>aci kockice"

Among the accelerator markers used so far, probably the best choice is tilde, ~ (used by OpenOffice). By examining various translation projects, it can be seen that tilde appears as non-accelerator^[20] two orders of magnitude less frequently than ampersand or underscore. A literal tilde would be written by doubling it, which makes it consistent with escaping of placeholder braces. Escaping would be processed in any ggx-format (or ggt-format) message, so that translators and translation support tools can always consider a lone tilde to be an accelerator marker. The above example would look like:

#, ggx-format
msgid "~Roll Dice"
msgstr "~Baci kockice"

and escaped literal tilde in a non-accelerated text:

#, ggx-format
msgid "Sample emoticon :) :( :~~( :0 :/ :P :| (* message."
msgstr "Primer poruke sa emotikonima :) :( :~~( :0 :/ :P :| (*."

Just like with tags, what the uniform accelerator marker resolves into would depend on the output destination. This would be defined when setting up the markup, per PO domain and target markup:

# Python/GTK+

def setup_markup ():

    ...

    markup_def(domain, "plain", "emph", ...)
    markup_def(domain, "pango", "emph", ...)
    markup_def(domain, "plain", "path", ...)
    markup_def(domain, "pango", "path", ...)
    ...

    markup_set_accel(domain, "plain", "_")
    markup_set_accel(domain, "pango", "")

    ...

To actually use normal tags for marking accelerators would not be good, because most strings which need accelerator marker otherwise need no markup. It is also better for consistency, lest different codes pick different tags, and better for translation processing tools, since they can with certainty throw out the accelerator marker without touching anything else. Finally, accelerator markers must also be processed in non-markup aware, ggt-format messages.

Of all the proposed extensions in this document that make up the generalized Gettext, customizable markup is by far the most demanding on programmers. For comparison, once argument capturing is swallowed, all other extensions feel conceptually the same as current Gettext. They may only differ slightly in appearance and provide more options, and some are even invisible to programmers (translation scripting).

In current Gettext, when a message is not translated in any of loaded language catalogs, one gets out exactly the string which was put in; the gettext call is effectively a no-op. With xgettext call this is no longer so, as the original string itself will be transformed according to markup. Programmers have to keep in mind to escape XML-special characters in plain-text looking messages. They have to make sure that messages are resolved into the appropriate (or at least acceptable) target markup for the given output destination. They should take care to substitute translation objects as arguments directly, rather than after converting them to native string.

To my taste, this is a small price to pay for being able to use semantic-like markup with different destination outputs, for not worrying if my obscure coding framework specific markup will confuse translators, for not thinking about escaping arguments when substituting them into strings which will be interpreted for markup, and for having ready-made means to validate markup in both source and translated messages. But tastes differ, and certainly there will be many programmers who dislike the added constraints that customizable markup brings with it.

The solution to keep everyone happy has already been mentioned: there would also exist the tgettext series of calls, which behave like xgettext calls in all aspects other than the markup, which they ignore. Messages in the PO file would have either ggx-format or ggt-format flag, depending on the call from which they were extracted. This removes any ambiguity for translators and translation processing tools. Consider the message:

msgid "Replace 'foo' with 'bar'."
msgstr ""

and imagine that the translator would like to use angled brackets around the "foo" and "bar" words. In ggx-format the valid translation would be:

#, ggx-format
msgid "Replace 'foo' with 'bar'."
msgstr "Zameni <foo> sa <bar>."

In ggt-format, < and > are not special characters, and the translation should be:

#, ggt-format
msgid "Replace 'foo' with 'bar'."
msgstr "Zameni <foo> sa <bar>."

The question that remains is how will the xgettext command decide which flag to put on a message. This is done by adding extra elements to call specification in the -k option: ,x for ggx-format calls, and ,t for ggt-format calls.

You may have noticed that, unlike all other proposed extensions, interface contexts as described so far are pure convention. They are in no way taken into account at run time, and the only "implementation" would be documenting available major and minor components. However, interface contexts can be optionally given one technical function.

In the section on customizable markup, it was discussed how to select the desired target markup when converting the translation object into a native string. Two ways were offered. The ideal way was automatic selection, by having smart output destinations (widgets, output functions) which themselves convert to native string and set the appropriate target markup. Since smart output destination are more likely not to be available, the other proposed way was manual selection. To recollect, for the three possible variants of conversion to native string, manual target markup selection went like this:

// C++/Qt

// Explicit conversion: target markup by the conversion method.
QString msg1 = xgettext("Processing <fn>{1}</fn>...")
                       .subs(filePath).toRich();

// Implicit conversion: target markup by a method before conversion.
QString msg2 = xgettext("Processing <fn>{1}</fn>...")
                       .setRich().subs(filePath);

// Immediate resolution: target markup by a dummy argument.
QString msg3 = ixgettext("Processing <fn>{1}</fn>...",
                         Gettext::Composer::Rich, filePath);

The third way to select target markup is by assigning the default target markup for each interface context. This makes sense because, for example, all @action:button texts in a program will be rendered by the same button widget, all @item:inlistbox by the same list box item widget, and so on. Target markups and interface contexts would be linked at the same place where the markup was set up:

// C++/Qt

void setupMarkup ()
{
    typedef Gettext::Composer GTC;

    const char *domain = "fooapp";

    GTC::markupDef(domain, "plain", "emph", ...);
    GTC::markupDef(domain, "qtrich", "emph", ...);
    GTC::markupDef(domain, "plain", "path", ...);
    GTC::markupDef(domain, "qtrich", "path", ...);
    ...

    GTC::markupSetClassPlain(domain, "plain");
    GTC::markupSetClassRich(domain, "qtrich");

    const char *plainUics[] = {"@action", "@option", "@label",
                               "@title", "@item",
                               "@info:status", "@info:shell", 0};
    GTC::markupSetUicPlain(domain, plainUics);
    const char *richUics[] = {"@info", 0};
    GTC::markupSetUicRich(domain, richUics);
}

Linking is done by the markupSetUicClass(domain, uicontexts) function for the particular target markup class. When an interface context is given only by the major component, that means to apply the target markup to all interface contexts with that major component and any minor component, except for major-minor combinations that were explicitly given. In this example, all major interface contexts except for @info were set to plain text class, as well as some particular @info:... contexts, and all other @info contexts were set to rich text class (which is here declared to be Qt Rich Text).

For this to work, interface context must be part of msgctxt message field, or else it could not be seen at run time. Translation object will check whether its context argument contains the interface context, and if so, it will try to set the target markup accordingly. If the given interface context was not linked to any target markup, target markup will remain at its global (domain) default. The above conversion example would now look like this:

// C++/Qt

// Explicit conversion.
QString msg1 = pxgettext("@info:progress",
                         "Processing <fn>{1}</fn>...")
                        .subs(filePath).toString();

// Implicit conversion.
QString msg2 = pxgettext("@info:progress",
                         "Processing <fn>{1}</fn>...")
                        .subs(filePath);

// Immediate resolution.
QString msg3 = pixgettext("@info:progress",
                          "Processing <fn>{1}</fn>...",
                          filePath);

Other than the added context, all conversion variants now look exactly the same as they would without markup in the picture (explicit conversion is done by the general .toString method, which respects any previously set target markup). Of course, target markup selected based on context can later be overridden by any of the manual selection methods.

When markupImport function is used to import markup from another domain, any context-markup links defined by that other domain would also be imported. So, by calling markupImport(domain, "gengettext"), one would at the same time get access both to the generalized Gettext's library built-in set of tags and to the built-in context-markup links:

// C++/Qt
void setupMarkup ()
{
    typedef Gettext::Composer GTC;
    const char *domain = "fooapp";
    GTC::markupImport(domain, "gengettext");
    GTC::markupSetClassPlain(domain, "plain");
    GTC::markupSetClassRich(domain, "qtrich");
}

Given that most appropriate defaults for target markup per target markup class would be defined for each coding framework binding (e.g. rich text would be set to "qtrich" in Qt bindings and to "pango" in GTK+ bindings) the absolute minimal example would be a single markupImport call:

// C++/Qt
#include <libintl_qt.h>
// ...coding framework specific bindings in use.

void setupMarkup ()
{
    Gettext::Composer::markupImport("fooapp", "gengettext");
}

Just like any of the tags can be overridden after an import from another domain, so can any of the context-to-markup links.

Translation scripting would introduce new syntax to the PO format: the msgscr field. This field would be manually added by the translator when necessary, to provide a scripted translation in addition to ordinary translation in msgstr field.

msgscr would contain text of the translation just as msgstr, but segments of the form $[callname arg1 arg2 ...] would be interpreted as scripting calls. This is similar to Unix shell command expansion syntax $(...), hence the approach is called the "PO shell". The scripting call consists of a call name followed by whitespace-separated list of arguments. Arguments can be literal strings, or message argument placeholders, which are substituted before dispatching the call. In the msgscr of this message:

#, ggx-format
msgid "No configuration available for {applet}."
msgstr "{applet} nema ništa za podešavanje."
msgscr "{applet} $[po-broju {applet} nema nemaju] ništa za podešavanje."

po-broju is the call name, {applet} is a placeholder which will be substituted to become a string argument, and nema and nemaju are two literal string arguments^[21].

If a literal string argument needs to have whitespace in it, it can wrapped in single quotes. If a placeholder is placed next to a literal string, the substituted value and the literal string will form a single argument. Everything in the substituted value is treated as literal text. It will not happen, for example, that a single placeholder substitution produces several call arguments because it contained some whitespace. To put it together, this call:

msgscr "... $[acall foo{bar}'baz qwyx'fum] ..."

has only one argument, no matter the value substituted for {bar}. Single-quoting does not prevent placeholder substitution. These two calls have the same single argument:

msgscr "... $[acall 'foo {bar} baz'] ..."
msgscr "... $[acall 'foo '{bar}' baz'] ..."

Scripting calls can be nested within scripting calls, for example:

msgscr "... $[if $[check-something ...] then-value else-value] ..."

A scripting call can be nested even at the call name position of the outer call, so that the string it returns is treated as the call name of the outer call.

A scripting call is considered properly executed if it returns a string (even an empty one). In that case its return value is concatenated to previously parsed segments, and parsing of msgscr continues. Once msgscr is parsed to the end, all collected segments (literals and call return values) are concatenated to produce the final translation of the message. If a scripting call returns something that is not a string or if it aborts, interpretation of the whole msgscr is aborted, and ordinary translation (msgstr) is taken as the final translation of the message.

Escaping is done as follows. The call start sequence $[ can only be escaped by inserting an empty call $[], giving $$[][. Empty call always returns empty string. This is preferable to introducing special escaping syntax because the probability of literal $[ in a message multiplied by the probability of scripting being needed for that message is extremely low. Within the call, closing square bracket and whitespace are escaped by putting them in single quotes, and single quote is escaped by doubling it. Braces, being placeholder delimiters, are escaped by doubling them (just like outside of a call).^[22]

Within msgscr, another argument placeholder is automatically made available: {0}. It evaluates to the ordinary translation, which may come in handy in some situations. For example, the set-property scripting call mentioned earlier is a bit magical, in that it is nowhere told on which string to set the property (unlike get-property, which is given both the string and the property name); set-property will automatically pick the ordinary translation as the string to set the property on. An explicit, non-magical variant set-property-of would be applied like this:

#, ggx-format
msgid "FooProcessor"
msgstr "FuProcesor"
msgscr "$[set-property-of {0} relative-case FuProcesoru]"

Sometimes it might be necessary that a placeholder substitution does not result in a string value for scripting call argument, but in a value of type "nearest" to the actual type of the substituted message argument. This would mostly apply to substitution of numbers, since as a string the number may have locale thousands separator and decimal comma, padding, and whatever else the coding framework provides, making it hard to convert it back to an integer within the scripting call definition. Nearest-type substitution is requested by adding the # extensions to the placeholder. For example, when the programmer knew that the substituted number will always be greater than 1 and therefore assumed that a plural call is not needed, the translation could be scripted^[23] like this:

#, ggx-format
msgid "{num} bytes"
msgstr "{num} bajtova"
msgscr "{num} $[plural-form {num#} bajt bajta bajtova]"

For precisely defining what "nearest type" means it must be known which types a scripting call can take, and that depends on how scripting calls would be defined. More on that in the section on sources of scripting calls.

When a plural message needs to be scripted, each msgstr[i] can be paired with a corresponding msgscr[i]. They would be ordered in the PO file as msgstr[0], msgscr[0], msgstr[1], msgscr[1]... Pairing would not be mandatory for all plural forms in the given message, since it may happen that not all of them need to be scripted.

There are several reasons why msgstr field would always be kept, rather than substituted with msgscr. Or, from the other side, why msgstr would not be made scripting-sensitive itself instead of introducing msgscr. One reason was already mentioned: in case the evaluation of msgscr fails, msgstr is used as fallback translation.

The second reason is that failure of msgscr does not have to be due to an error on the part of translator, but may be by design. For example, in:

msgscr "... $[get-property fooprop {bararg}] ..."

it may be that only some arguments have the requested property set, and the scripted translation applies to them, where for other arguments the ordinary translation is fine. A more extreme example is property setting itself:

#, ggx-format
msgid "FooProcessor"
msgstr "FuProcesor"
msgscr "$[set-property relative-case FuProcesoru]"

where the scripted translation always fails, as the call is executed for side-effect of setting the property on ordinary translation.

The third reason is configurability. As will be shown later, translation scripting requires a scripting language interpreter to run within the process, which in some special environments may be too heavyweight or considered a security risk. Generalized Gettext library may then be compiled without scripting support or the interpreter may be globally disabled, and ordinary translations will still be available.

The fourth, final reason for always keeping msgstr is backward compatibility: if needed in some context, a compiled PO file (MO file) with scripted elements can be used with classic Gettext calls. This would be done by making two MO file entries for each scripted message. One would be an ordinary entry, composed of msgid value and msgstr value, as if msgscr weren't present. The other entry would be msgid value with a special tail (e.g. one '\x04' character^[24]) and msgscr value. This would make a classic Gettext call succeed on ordinary entry, while a generalized Gettext call would look for both entries.

These reasons notwithstanding, it could actually be allowed to remove msgstr and leave only msgscr on a message. This is because the underlying assumption so far was that only a very small number of messages (say less than 1%) will need to be scripted. If this assumption would not hold, if a significant portion of messages, especially longer messages, would need to be scripted, than the great duplication of text would probably trump all the proposed advantages of keeping the msgstr. One of Gettext's tools would provide to convert all msgstr to msgscr in a given PO file; msgmerge would add msgscr instead of msgstr on new messages if it would detect that the majority of old messages have only msgscr. If a scripting call aborts, the script language interpreter is disabled, or the MO file is used for classic Gettext calls, messages that had only msgscr would simply appear as untranslated.

Scripting calls useful for many languages, such as set-property and get-property, could be implemented within the generalized Gettext library and in its core programming language, i.e. C (or C++). This could also be the done for calls which are specific to one language, but widely useful within it (in many PO domains). Language-specific calls would live in separate namespaces, so that the same call name would have different behavior in different languages.

Stopping at this, however, would lead to two problems. The first problem is that C/C++ may be tedious for adding and testing new calls, requiring building and running development version of the library. This is especially problematic for language-specific calls, where the maintainer of the generalized Gettext library may not understand the exact language-specific functionality needed. The second problem is more important: it would not be possible to define scripting calls specific to a certain PO domain. This is something that a translator with basic programming skills might want to do, and there is a non-negligible number of such translators in the free software translation environment. Given the usual amount of cooperation in free software projects, even a translator without any programming skill might find someone to implement a special scripting call.

It should therefore be technically and organizationally simple to add and modify scripting calls, both within the generalized Gettext library and per PO domain.

To achieve the technical simplicity, generalized Gettext library would provide a scripting language for defining scripting calls, and only those calls that need high performance and controlled memory use would be implemented in C/C++. This scripting language should be a well-known general-purpose language, rather than a custom one. It should be simple to program in, lightweight in terms of memory footprint and interpreter runtime dependencies, and designed for sandbox operation. Two candidates come to mind: ECMAScript (better known through JavaScript as its main dialect) and Lua.^[25]

Lua is the smaller of the two, with a MIT-licensed implementation designed for embedding. It is particularly frequently used for scripting in computer games. Unfortunately, Lua is too small. The biggest problem is that it does not have a Unicode string type, but treats strings as raw byte arrays. An external string type would have to be provided instead, which would look rather dirty. Another problem is that it defines its own pattern-matching microlanguage, rather than using POSIX/Perl regular expressions.

This leaves us with JavaScript, which is quite popular for scripting web pages, and is used more and more as the embedded scripting language in larger programs and environments in general. There are several free implementations of the basic JavaScript interpreter, like SpiderMonkey (MPL/GPL/LGPL-licensed), QtScript (LGPL-licensed), or KJS (LGPL-licensed).

For scripting calls provided by the generalized Gettext library, it is not very important where their definitions would reside. There would obviously exist one directory for general calls, and one directory per language for language-specific calls; everything beyond that is code layout detail. It is much more important to decide where calls per PO domain would be placed, because that determines the organizational simplicity of implementing domain-specific calls.

From the installation tree point of view, the proposition is to introduce a separate root directory for scripting call definition files at the same level as LC_MESSAGES of the given language, called LC_SCRIPTS. This root directory would contain one directory per each PO domain that has some custom scripting calls defined, and within that directory a file named main.js. For example, if the PO domain is fooapp, the installation tree would look like this:

$PREFIX/share/locale
    aa/
        LC_MESSAGES/
            fooapp.mo
        LC_SCRIPTS/
            fooapp/
                main.js
    bb/
        LC_MESSAGES/
            fooapp.mo
        LC_SCRIPTS/
            fooapp/
                main.js
    ...

The reason for having a domain-named subdirectory instead of a single file (fooapp.js) is that, in general, more than one scripting file, or files other than scripting files, may be necessary for a single PO domain (an example of this will be shown later). This subdirectory would be called a scripting module

As for the source tree of a Gettext-using program, scripting modules could be placed like this:

fooapp/
    doc/
    po/
        LINGUAS
        aa.po
        bb.po
        ...
        scr/
            aa/
                main.js
            bb/
                main.js
            ...
    src/
    ...

Another possibility is to collect scripting modules into another top-level subdirectory, say poscr/.

When JavaScript is used for scripting web pages in a browser, it is extended with a "browser object model" (BOM), a set of global objects for accessing the web page source (HTML code) and browser internals (shortcuts, tabs, etc.) Similarly, we have to define a "translation object model" (TOM) through which JavaScript code in a scripting file can connect with the PO shell.

The proposal is to have the following global objects:

As an example of employing the TOM, recall the set-property call from previous examples, which set a property on the string given by msgstr field. Here is how this call could be defined, to set an arbitrary number of properties:

/* Set properties of the phrase given by msgstr.
 * Always signal fallback.
 */
function set_msgstr_property (/*...*/)
{
    if (arguments.length % 2 != 0)
        throw Error("Property setter given odd number of arguments.");

    phrase = message.msgstr();

    for (var i = 0; i < arguments.length; i += 2)
    {
        var prop = arguments[i];
        var value = arguments[i + 1];
        domain.set_phrase_property(phrase, prop, value);
    }

    throw gtcore.fallback();
}
domain.set_call("set-property", set_msgstr_property);

Every function in JavaScript is variadic, in the sense that when the function is called, any declared parameters in the header will get arguments assigned to, and both those assigned and any remaining call arguments will be accessible through the arguments array. Therefore here the /*...*/ comment was put into the header to indicate intentional variadicity. The first thing is to assure that there is an even number of arguments (property keys followed by values), or else throw an exception. Then, the phrase on which the properties are set is taken as current message's msgstr field, from the message object. Within the property setting loop that follows, domain.set_phrase_property method is used to link the phrase and property key and value in the current PO domain. Since set-property is a side-effect scripting call, at the end an exception is thrown to signal the fallback to msgstr. But, if a JavaScript Error object would be thrown, that could register as a warning somewhere, in a debug mode or so; instead, a special object produced by gtcore.fallback method is thrown, which causes the fallback to be silent. After the function body, domain.set_call method is used to link the set_msgstr_property function name to the PO shell set-property call name.

The corresponding get-property method would be defined more simply as:

/* Get value of the phrase property.
 * If the property is not set on the phrase, signal error.
 */
function get_phrase_property (phrase, prop)
{
    value = domain.get_property(phrase, prop);
    if (value == undefined)
        throw Error(gtcore.compose(
            "Phrase '{1}' does not have the property '{2}' defined.",
            phrase, prop));
    return value;
}
domain.set_call("get-property", get_phrase_property);

The only interesting thing here is the use of gtcore.compose method to assemble the error message. JavaScript lacks a standard string formatting method, so gtcore object would provide a reduced version of generalized Gettext string composition functionality.

Call names given to domain.set_call can be any Unicode strings, so translators may decide to use language-specific call names (including non-Latin scripts) to blend in more elegantly into the translation in the PO file. There is no standard way to specify the encoding of a JavaScript source file, so it would be required that scripting files are UTF-8 encoded.

Since scripting calls and underlying JavaScript functions would be found in the generalized Gettext library and in various PO domains, the questions of modularity and scoping arise.

Scripting file modularity comes into picture when in the current scripting file one wants to use a definition (function, object, variable) from another scripting file. The other scripting file may be within the same PO domain, in another PO domain, or in the generalized Gettext library; it may also belong to another language^[26]. JavaScript itself does not define any kind of module system, so there is no convention to follow. The proposition is to have a single gtcore.import method, which would work as follows. The function afunc would be imported from file afile.js in the current PO domain with:

gtcore.import("afile", "afunc")

This call would make afunc available for calls anywhere in the current scripting file, no matter in which scope it would be executed. gtcore.import would actually take any number of names to import:

gtcore.import("afile", "afunc1", "afunc2", "AnObj3", ...)

To import from a file within a subdirectory of the current domain, dotted path would be used:

gtcore.import("asubdir.afile", "afunc")
gtcore.import("asubdir.asubsubdir.afile", "afunc")

Importing from another domain would work by prepending the domain name with a colon:

gtcore.import("adomain:afile", "afunc")
gtcore.import("adomain:asubdir.afile", "afunc")

To import from another language, language code and percent sign would be prepended to domain or file path:

gtcore.import("alang%afile", "afunc")
gtcore.import("alang%adomain:afile", "afunc")

Importing from the generalized Gettext library would be done in the same way, by supplying its domain name gengettext.

One problem here is that Gettext has no mechanism of searching through various locale directories for domains. Instead, the locale directory of a program is configured at build time, making it hard-coded in the installation. This presents a problem for importing definitions from domains other than the current and gengettext (since generalized Gettext library would bind its own locale directory), when programs are installed in different prefixes. One solution could be to introduce something like GETTEXTPATH variable, which would take precedence over build-time bound locale directory. In my opinion this would be a welcome addition even to ordinary Gettext^[27], but it could be limited only to use within generalized Gettext (it would also be needed for domain resolution in connection with customizable markup).

Scripting call scoping refers to the mechanism by which the underlying function would be selected for a call name parsed from msgscr string in a PO message. Obviously, if the PO domain to which this message belongs has its own scripting module, then any domain.set_call in it would have top priority:

domain.set_call("a-call", a_call_func)

Technically it would be possible to stop here, and rely on the scripting file modularity for the rest. For example, to set a call to a function from generalized Gettext library, one could add this in domain's scripting module:

gtcore.import("gengettext:main", "a_call_func")
domain.set_call("a-call", a_call_func)

But this would imply that whenever a translator wants to use scripting in the PO file, even if core scripting calls (those from the generalized Gettext library) would be completely sufficient, a scripting module with lines such as above would have to be added. Since most of the time core calls should be sufficient, having to add domain-specific scripting modules all the time would be too burdensome.

Instead, there would be gtcore.set_global_call method, which would make the call available in any PO domain within the program process, but limited to language of the message (i.e. no global calls across languages). Global call setting should be used sparingly, because it would mask dependencies between PO domains. It would most prominently be used in scripting modules of the generalized Gettext library, but it could also be reasonably used in base libraries of various coding frameworks. When a call name is seen in the PO message, first domain-specific calls would be checked for a match, and then global calls.

Another possibility could be to explicitly import another domain's calls through a PO header field:

msgid ""
msgstr ""
...
"Scripting-Call-Source: foodomain, gengettext;\n"

The order of domains would determine the scoping order, in cases when two domains set a call with the same name. This is much less burdensome than adding a scripting module for the sole purpose of importing calls, but it still requires that the translator remembers to do it. It also has the same issue with locating locale directories as mentioned above.

Once translation scripting is available, two interesting "second-order" possibilities appear: filtering messages and runtime contexts.

Consider the UI scenario where a menu item expands into a submenu, such that the parent menu item and submenu items imply a continued phrase. Two examples:

File
    -> New
    -> Open
    -> Open With
        -> FooViewer
        -> BarShow
        -> ...

(context-menu)
    -> Open
    -> Cut
    -> Create New
        -> Folder
        -> Text File
        -> Application Link
        -> ...

Implied phrases here are e.g. "Open With FooViewer" or "Create New Text File". In languages with case declension, the submenu items would probably look better if put into a non-basic case. If submenu items are also used elsewhere where the basic case is appropriate, it is not possible to simply translate them using the appropriate non-basic case. With scripting, as we have seen, items could be given properties:

msgid "Application Link"
msgstr "Veza do programa"
msgscr "$[set-property object-case 'vezu do programa']"

But, where would that property be fetched? Normally, translated items would be simply inserted into the "Create New" submenu. The solution is to insert items through a filtering message, which consists of a single placeholder and an appropriate context:

// C++/Qt
foreach (const QString &ft, creatableFileTypes) {
    QString ft1 = pxgettext("@item:inmenu Create New",
                            "{filetype}")
                           .subs("filetype", ft).toString();
    createNewFtMenu->addAction(ft1);
}

or in the PO file, with scripted translation:

msgctxt "an item in Create New -> ... menu"
msgid "{filetype}"
msgstr "{filetype}"
msgscr "$[get-property object-case {filetype}]"

Filtering messages would normally be added on translator's request. It would never be a problem to add a filtering message, even when it helps only one language; from the code viewpoint it would be a no-op, and from the translation viewpoint trivial to translate. Filtering messages can be added even during a message freeze, since they do not introduce any new text.

In the two examples above, the UI context was such that a phrase segment (a file type) was inserted into a static implied phrase ("Create New ..."). In other words, the only variable part of the implied phrase was the segment being inserted. However, some UI contexts can lead to implied phrases whose parts depend mutually on one another. For example, in a file search dialog there can be options such as this:

[ ] Find all files created or modified:
    ( ) between ...
    ( ) during the previous ____n<> [hours|days|...]

Here [ ] denotes a checkbox, ( ) a radio button, ____n<> a number spinbox, and [hours|days|...] a listbox with those items. Already with current Gettext, the programmer can properly pluralize listbox items, by providing that they are retranslated whenever the number in the spinbox changes. The same can be done with "during the previous" text label in front, because the adjective "previous" may need to match the plurality of the noun in listbox. But, the adjective "previous" may also need to match the grammatical gender of the noun in listbox. Neglecting pluralization for brevity, if the PO message of this text label is:

msgid "during the previous"
msgstr ""

there is no link with the noun in the listbox, and therefore no way for translator to script conformance to noun's gender.

To establish the missing link, the translator can ask the programmer to retranslate this message on every change of the listbox item, and to set on that message a runtime context which indicates the current item in the listbox. A runtime context is simply a pair of two strings, the context keyword and runtime value. It would be added by the inContext method of the translation object:

// C++/Qt
QString prLabel = pxgettext("runctxt: period=[i|h|d|m|y]",
    // TRANSLATORS: This message has the runtime context 'period'
    // with values: 'i' for minutes, 'h' for hours, 'd' for days,
    // 'm' for months, 'y' for years.
                            "during the previous")
                           .inContext("period", periodKey).toString();

The message has a comment informing translators of the runtime context, its keyword and possible values, but also a semi-formal disambiguation context which succinctly states the context keyword and values. This is necessary because when a runtime context is modified or added to a message, translators must unconditionally review that message (same as when the ordinary context is added).

The translator can now script the translation as follows:

#. TRANSLATORS: This message has the runtime context 'period'
#. with values: 'i' for minutes, 'h' for hours, 'd' for days,
#. 'm' for months, 'y' for years.
msgctxt "runctxt: period=[i|h|d|m|y]"
msgid "during the previous"
msgstr "tokom poslednja"
msgscr "tokom $[by-context period i poslednja ... y poslednje]"

by-context call would take the runtime context keyword followed by pairs of context value and string to return for it. This call could be defined as (omitting some bells and whistles):

function select_by_message_context (ctxtkey /*...*/)
{
    var ctxtval = message.runctxt(ctxtkey);
    for (var i = 1; i < arguments.length; i += 2) {
        if (ctxtval == arguments[i])
            return arguments[i + 1];
    }
}
domain.set_call("by-context", select_by_message_context);

The new element here is the use of the message.runctxt method to fetch the value of the runtime context with given keyword. Note that if there is no match, this function will silently "fall off" the end; in JavaScript this means that the return value will be undefined. Since a valid scripting call must return a string, returning undefined will automatically cause fallback to ordinary translation.

Not all text shown in a Gettext-based program comes from PO catalogs at run time. One frequent source of such text are .desktop files, which store basic information about a program, such its name, version, execution pattern, etc. Information from installed .desktop files is read by other programs, such as program launchers to list available programs for starting, or file managers when offering programs to open a file. Here is an excerpt from a .desktop file:

[Desktop Entry]
Exec=planner %F
Icon=gnome-planner.png
Type=Application
# ...
Name=Project Management
Name[am]=የዕቅድ ጉባኤ
Name[ar]=ﺇﺩﺍﺭﺓ ﺎﻠﻤﺷﺭﻮﻋ
Name[be]=Кіраваньне праектам
Name[bg]=Управление на проекти
Name[ca]=Gestió de projectes
Name[cs]=Správa projektů
Name[da]=Værktøj til projektstyring
Name[de]=Planner Projektverwaltung
# ...

You can observe that translated entries are stored within the file itself. When a program reads an entry from a .desktop file, it will automatically fetch the translation into current user's language (if there is one). But, since these translations do not come from a compiled PO file, there can be no scripted translation to set properties on them (declension forms, gender, etc). How to work around this?^[28] For this we need some ingredients.

The first ingredient are property maps, or pmaps for short. Somewhat similar to the previously mentioned domain.set_phrase_property(phrase, prop, value) method to set a single property on the given phrase, there would exist a domain.load_phrase_properties(filename) method to read an arbitrary number of phrases and their properties from a text file. This file is called a property map, and it would have a very simple format:

# cities.pmap
=:Athens:Atina:nom=Atina:gen=Atine:dat=Atini:acc=Atinu::
=:Paris:Pariz:nom=Pariz:gen=Pariza:dat=Parizu:acc=Pariz::
...

Each phrase entry would start with two characters. The first character (here =) would be taken as the key-value separator in a property, and the second character as the separator of properties (here :). A property without a key (i.e. without a key-value separator) would be understood as the phrase itself. There could be more than one phrase, e.g. to cover alternative spellings, or as in this example, in case at run time a non-translated name is received for whatever reason. The entry would be terminated by double property separator. Newline would have no special meaning, so it could be used within phrases and property values. Comments could be written after # placed before an entry start (within an entry, # would have no special meaning).

The second ingredient is the realization that, although an installed .desktop file carries its own translations, it was likely translated through a PO file. This works by applying a tool which extracts translatable entries from .desktop files into a POT file, and then takes the translations from PO files and inserts them as entries back into .desktop files (insertion can happen at various moments, for example at build time). In this way translators can keep using the standard PO workflow and tools, instead of having to adopt another workflow and tools for every special file format that comes along. So, in one or the other PO file, translators will see the Name= entry from the .desktop file snippet above as:

#: ../data/planner.desktop.in.in.h:2
msgid "Project Management"
msgstr ""

Putting the two together, the translator can set properties by adding a pmap entry to a translator comment in the PO file:

# pmap: =:gen=Upravljanja projektom:dat=Upravljanju projektom:
# pmap: =:acc=Upravljanje projektom:ins=Upravljanjem projektom:
#: ../data/planner.desktop.in.in.h:2
msgid "Project Management"
msgstr "Upravljanje projektom"

The comment starts with pmap: keyword, and then an almost normal pmap entry follows. The only difference to an entry in a standalone pmap file is that the phrase (no-key property) is missing, and that is because the phrase is the translation itself. In order not to have to put a very long comment line, any number of pmap: comments can be given, and all the defined properties will be unified. A tool would be made to go through a PO file, collect all pmap entries, and write them in a pmap file. This generated pmap file would be put in the scripting module of the PO domain, and the main.js file in there would contain a domain.load_phrase_properties call to load the pmap file.^[29]

You may have noticed that this actually does not solve the problem with .desktop files. It works only when the extracted pmap entries are used in messages found in the same PO domain. There are situations when this is the case, like when a program uses custom files for storing its internal text data together with translations. But the point of .desktop files is that data in them is used across programs, i.e. in messages in various PO domains. On the technical level, this can be handled by putting generated pmap files into a designated system location and loading them in the scripting module of the generalized Gettext library itself, through a global gtcore.load_phrase_properties call. Then, when a property is looked up by a scripting call in a certain PO domain, first domain-specific property storage would be checked, followed by the global storage. To do this on the organizational level (distribution, packaging) should not be too hard either. Programs would simply install generated pmap files into the designated location, which they would obtain from the configuration of the generalized Gettext library (e.g. through pkg-config). There are few details to work out here (e.g. the designated location, could be within the gengettext scripting module or elsewhere), but nothing of conceptual difficulty.

One concern may be what would happen to existing program sources once generalized Gettext is introduced. Most importantly, there would be no backward incompatibility and no current feature of Gettext would become deprecated. So, simply, all existing programs could continue to use current Gettext, and new programs could be written in that way. Due to strict format separation, current and generalized Gettext calls could even coexist in the same program, for example, to facilitate incremental conversion.

If the maintainers of an existing program wish to fully and immediately convert to generalized Gettext, that should not be too hard either. A script could be written to handle the majority of cases, which are either no-argument messages, or simple argument substitutions. The minority of special cases, e.g. where arguments are substituted at a later point, can be reported by the script and handled manually. I had done something like this for the KDE 4.0 release, switching all the sources in KDE repository (about three million lines) to a new translation call syntax. It took me several days to make the conversion script and test its behavior, and several more days to manually convert special cases.

It has already been mentioned that the generalized Gettext library would have to have well-fitted bindings for various coding frameworks, and that these bindings could be provided by those coding frameworks themselves. Going one step further, this whole document could be understood as proposing a specification rather than implementation. There could be several generalized Gettext providers, implemented as a layer above the Gettext proper. For example, one such implementation could be based on QtCore and QtScript, and another on Glib and SpiderMonkey. This may look like wasted effort, but the implementation should be rather lightweight if the foundation libraries provide all the needed bits: a string type, a few container types, IO streams, locks, an XML parser, regexes, and a JavaScript interpreter. Therefore the advantage of not introducing another low-level dependency chain into the stack may be more important than having a single implementation.

Another reason for implementing generalization as a layer above Gettext could be to test and polish all the features in the real world use, and then introduce the refined result into the Gettext itself.

Some support from Gettext side could be provided outright, though. For one, activation of gg*-format flags through extra elements in the -k option of the xgettext command should not be a big problem, since this does not interfere with any other functionality of xgettext. The flags should perhaps be changed somewhat, e.g. to xgg*-format, to leave the original suggestions for the time when the generalization would become part of Gettext itself.

More problematic would be the msgscr message field. It would be strange to introduce a new PO field into Gettext which is not used by Gettext itself. On the other hand, this would make sense if the intention would be to make one implementation a part of Gettext at some point. If the msgscr field is not introduced, it would be possible to write it within the msgstr field, with a special separator, here |/|:

#, ggx-format
msgid "Events on {date}"
msgstr "Događaji na {date}|/|Događaji $[on-date {date}]"

While this may be "dirty", it is much less so than the earlier widespread approach of embedding the disambiguation context (before msgctxt was introduced), for example:

msgid "application name|Unknown"
msgstr "nepoznat"

This is because a translator who knows nothing about scripting would have nothing to mess up, unlike with embedded context which unaware translators happily translated along with the text. The separator should be chosen a bit more verbose and unlikely to be a literal part of the text, like |/| is above, in order to practically never have to escape it. If the separator is chosen in this way, it can even be required that the original text must not contain that sequence.

Other than the format flags, the call specification elements, and the msgscr field, there is nothing else that current Gettext would really have to support. Everything else would be part of the generalization layer.^[30]

PO files are used to translate just about everything, not only user interfaces of programs that use Gettext calls at run time. There are two types of non-native sources, dynamic and static. Dynamic sources are programs which use a system other than Gettext to translate their user interface. Examples of these are programs within Mozilla and OpenOffice^[31] suites. Static sources are any kind of text meant to be read standalone and sequentially, such as program documentation or (in some instances) web pages.

Non-native sources are translated over PO files with the help of an extractor-injector tool. This tool first splits all text in the source file into distinct segments, and extracts the segments as PO messages into a POT file. The POT file is used in the PO translation workflow as usual, resulting in translated PO files. The tool then takes the original document and the translated PO file, and injects translations in place of (or in addition to) original text in a copy of the source file. This produces the final translated source file. With some tweaks to the extractor-injector, all non-native sources, dynamic or static, can be represented through generalized Gettext PO messages.

For dynamic sources, the extractor-injector should be made to convert any native argument placeholders (or formatting directives) into uniform placeholders, and any native markup into customizable markup. For example, if PO messages from a non-native source would currently look like this:

#: src/minimap.cpp:90
msgid "Could not get image for terrain: $terrain."
msgstr ""

#: data/core/encyclopedia/drakes.cfg:26
msgid ""
"The time that passes between one <ref>dst=egg text='egg'</ref> "
"laying to the next."
msgstr ""

the extractor-injector could be upgraded to convert and back-convert placeholders and markup, to produce proper generalized Gettext messages:

#: src/minimap.cpp:90
#, ggx-format
msgid "Could not get image for terrain: {terrain}."
msgstr ""

#: data/core/encyclopedia/drakes.cfg:26
#, ggx-format
msgid ""
"The time that passes between one <ref dst='egg'>egg</ref> "
"laying to the next."
msgstr ""

Customizable markup is the key here, since it enables straightforward mapping of native markup. If the generalized Gettext markup were fixed instead, it would be hard, ugly, or even impossible to meaningfully map native markup.

If a native argument placeholder contains a formatting sequence, that sequence can simply be dropped on extraction, and put back on injection. For example, if the native string were "Total ~,2@f kg of iron loaded.", the converted string would be "Total {1} kg of iron loaded.". Of course, in peculiar situations where exposing formatting sequences to translators would be preferable, that too is possible: "Total {1:,2@f} kg of iron loaded.".

Static source files are normally written in some sort of a formal markup language, like Docbook, or semi-formal markup, like a wiki. For non-XML based markup, it would be necessary to fully map it to an XML-like tags, but that should be straightforward. In fact, this approach is nothing new: I have seen at least two instances where a non-XML markup was converted to Docbook for translation. The tag freedom of customizable markup of generalized Gettext can only make this procedure easier.

XML-based static source formats, like Docbook or Mallard, would simply remain themselves. Only two small tweaks would be introduced.

The first tweak concerns custom XML entities, such as:

#. Tag: phrase
#: index.docbook:121
msgid "A typical &kgoldrunner; game"
msgstr ""

Generalized Gettext customizable markup does not allow custom entities, for the reasons outlined before, so there would be two ways to go about this. The first way is to expand entities on conversion:

#. Tag: phrase
#: index.docbook:121
#, ggx-format
msgid "A typical <application>KGoldRunner</application> game"
msgstr "Tipična partija <application>K-zlatobojca</application>"

Expansion is the right thing to do for most of the entities. This is because translators may need to modify the text behind the entity, either to add some grammar modifications (such as case endings on nouns), transcribe the text another script (such as Cyrillic), outright translate it, or any combination of those (such as above: translation with case ending). If expansion is technically difficult or a large chunk of static text is inserted^[32], the entity should be replaced with a uniform placeholder:

#. Tag: phrase
#: index.docbook:121
#, ggx-format
msgid "A typical {kgoldrunner} game"
msgstr "Tipična partija {kgoldrunner[K-zlatobojca]}"

The translation of this message introduces another feature of uniform placeholders: the [value] extension. This extension allows the translator to override the original argument that would have been substituted with an arbitrary value. Bracketing is necessary because the value could contain a character which starts another extension, such as !, :, etc. Literal square brackets would be escaped as usual, by doubling.

The second tweak is about custom elements that some extractor-injectors insert into text to represent cut-out portions of the text. For example, if a Docbook paragraph containing a footnote:

<para>Looking at [...] a preview image for each individual person
<footnote><para>This of course also applies to places, keywords,
and other [...].</para></footnote> as can be seen in [...].</para>

would be extracted with xml2po command, two PO messages would appear:

#: browsing.docbook:53(para)
msgid ""
"<para>Looking at [...] a preview image for each individual person"
"<placeholder-1/> as can be seen in [...].</para>"
msgstr ""

#: browsing.docbook:54(para)
msgid ""
"This of course also applies to places, keywords, and other [...]"
msgstr ""

Here <placeholder-1/> is the custom element that represents the place where the footnote was cut out of the main text of the paragraph. Custom elements are not good for validation, since then the validator has to know not only about the normal markup, but also about what a particular extractor-injector could add. This is simple to solve, by using a uniform placeholder instead:

#: browsing.docbook:3(para)
#, ggx-format
msgid ""
"<para>Looking at [...] a preview image for each individual person"
"{placeholder-1} as can be seen in [...].</para>"
msgstr ""

Another requirement is that the extractor-injector produces messages which contain well-formed XML on their own, but that is quite natural, and all the extractor-injectors that I know of already do so.

At the end of 2003, several months after I started to work on PO based translations, it had occurred to me that the grammar inflection of my native language could be handled with some simple translation scripting. I quickly put up a proof of concept, that can still be seen here: http://nedohodnik.net/misc/cotras-intro.html. It completely ignored Gettext as such; it was something I wanted to see working just for fun, without a clear intention of going that way. But I did show the concept to others, translators on the KDE translations mailing list. At that point I was given the first practical clue (by Federico Cozzi): yes, looks interesting, but it really should be based on Gettext and PO files.

About half a year passed without me doing anything further about this, but I was slowly getting to know how the KDE base libraries used Gettext internally, and how the KDE code was developed and released. I realized that, since KDE had an internal wrapper layer around Gettext calls instead of using them directly, it was indeed possible to add translation scripting as a "post-processing" step, after the Gettext call and before the translation is send further away. So by the late 2004 I had made a real proposal, including test implementation and performance measurements, to add such a layer in KDE 4 (at that time KDE 3 was the actual major version). It was necessary to wait for KDE 4, since argument capturing would break the binary compatibility of the existing translation layer.

Again I run the proposal through fellow translators, and got more advice. Most importantly, my proposal used s-expression syntax for piecing up the complete message, and I was told (by Krzysztof Lichota) that it would look less menacing if it used string interpolation calls. An interpolation call as described in this document is an s-expression in itself, but localized to only what needs to be scripted, rather than taking over the complete text. At that time I also intended Guile to be the scripting language, but I was given a strong warning (by Stephan Kulow) that a general-purpose language, which can reach into the system, would be unacceptable from at least the security standpoint. The last version of that proposal, after a few updates during 2004 and 2005, remains at http://nedohodnik.net/misc/ktranscript.html (other than being a historical item, there is also the section "Performance Considerations", with test cases and performance measurements to show that runtime performance of a translation scripting system is quite sufficient).

More time passed, and by the end of 2005, things started to develop. KDE code was branched for what was to become KDE 4.0 two years later, and I had a good grasp of how the KDE translation system worked internally. It was using Gettext in a bit custom way, e.g. having own syntax for plural and disambiguating contexts, and a patched version of the xgettext command was needed to extract messages. Around that time msgctxt was introduced to Gettext, so I took up to convert KDE base library and applications to work with standard Gettext. That enabled me to prepare a slot where translation scripting layer could be inserted, should it be accepted at some point in the future.

This included switching translation calls to argument capturing. This was nice to have even without scripting, since it made possible automatic locale-formatting^[33] of arguments, and translation calls became shorter. The latter was due to the fact that two variants of conversion to native string (QString) were provided, explicit conversion and immediate resolution. Implicit conversion was omitted, just-in-case. Combination of explicit conversion and immediate resolution calls worked nicely ever since, without anyone complaining (or at least not loudly enough) about lack of implicit conversion. I had considered some other approaches too, but various people have helped eliminate them (e.g. Nicolas Goutte was particularly involved into that aspect).

By spring 2006, the switch to standard Gettext and new translation call syntax was completed. With KDE 4.0 release not yet nearly in sight, I decided to wait a bit before inquiring again about translation scripting. I did this near the end of 2006, again taking Guile for the interpreter, and trying to back up with some security features (e.g. ignoring scripted translation when process is running under root permissions). Nevertheless core developers were strongly opposed, but, luckily, there was a workable option. In the meantime, the JavaScript interpreter of the Konqueror web browser, KJS, became part of the KDE base libraries. Now there was an interpreter which was both a part of KDE base libraries and of the sort intended to run in a sandbox.

So, JavaScript it was, and in the first half of 2007, I added the translation scripting layer into the prepared slot. At that time I wasn't really thinking of doing this on the level of Gettext, since it was a highly experimental attempt. Scripted translation was written into msgstr field, after the |/| separator; there was no means of escaping, the text was simply forbidden from containing this sequence. This too worked nicely ever since. There were no cases where an inexperienced translator messed something up, nor any other ill effect.

Translation scripting as implemented at that time basically hasn't changed since. Some functions were added as needs arose (e.g. handling of property maps), and that was it. The single biggest change was adding the possibility to have scripting calls within scripting calls. It is interesting how this came about. Originally I intended for "call within a call" scenario to be handled by defining a new specialized call in the scripting module for the PO domain that needed it. But, in discussion with translators (notably Yukiko Bando), it became apparent that it would be really convenient to have as many as possible scripting calls provided by the KDE base libraries scripting module (which stood for generalized Gettext library as proposed in this document), and introduce domain-specific scripting modules only for truly exceptional cases.

The translation scripting system proposed by this document is generally the same as the one currently in KDE, except that it is much stricter with respect to namespaces. In the current KDE system, all translation-related extensions to JavaScript are methods of a single object (called Ts) instead of being thematically separated into different objects as proposed now (message, domain, etc). Also, all property setting on phrases is global, meaning that properties set in the program's PO domain can clobber those in an underlying library's PO domain. None of this has created a single problem in practice so far, but I considered it prudent to tighten this aspect in the current proposal.

In the latter half of 2007, some programmers expressed a wish for a way of semantically marking up elements in messages. They were fed up with thinking whether to put a quote, bold face, or whatever else, around e.g. a file name, and wanted instead to simply say "this is a file name". We considered possibilities of doing this outside of the string, in argument substitution methods, but all ways turned clunky. They would have also taken markup out of the translator's sight, which was a pity since semantic markup also provides some context. Therefore we came to the approach of having XML-like semantic markup directly in strings. Selection of target markup (when converting to native string) was done only through interface contexts, and in no other way.

Unlike translation scripting, however, semantic markup did not work that well. I could see programmers frequently struggling to choose the right tag from the provided fixed set, even if that set was "carefully chosen" (discussed with translators, etc). From the technical side, a translation object was not made to substitute another translation object as an argument directly, so markup was always resolved early. This led to problems with wrong selection of target markup, and automatic escaping could not be done. This in turn caused artifacts due to (lack of) escaping. It is not that there were big problems in practice, but the details were ugly enough, and fixed tag set too limited. It was also allowed to use Qt Rich Text markup alongside semantic tags, and most programmers simply continued using only those (which would be just fine in the proposed customizable markup approach).

Although interface contexts were presented as an element of semantic markup, they were used much more than actual in-string markup. Programmers were never uncomfortable with adding interface contexts on request (or when known "i18n fixers" would commit contexts themselves), and some maintainers have even equipped most of their programs with interface contexts. No translators have complained about the increased workload (which is, as shown earlier, on the order of 3% by word count in PO files highly equipped with interface contexts). The proposed set of major and minor components of interface context in this document was carried over from the current KDE system without modifications. This set has almost never changed since inception (only :row minor component was added later), indicating its stability.

I have mentioned a few people by name above, but many others have given valuable advice (and reality checks) along the way. To name a few more: Harri Porten and Maksim Orlovich, core developers of KJS, have given me a hand with integrating KJS as the JavaScript interpreter; Bruno Haible, the maintainer of Gettext, has added support for KDE 4 translation system into Gettext tools; Oswald Buddenhagen, the maintainer of Qt Linguist system, has discussed with me a lot about what we could do for translations in KDE/Qt 5. That discussion prompted me to write this document.

As can be seen from the history overview, this proposal grew out of the original drive for introducing translation scripting. Other people had similar ideas.

In 1998, Sean M. Burke had a brush with plural handling, for which Gettext had no support at the time. He realized the full implication and proposed a function-based replacement for Gettext, called Maketext, for use in Perl programs. His text can be found at http://search.cpan.org/dist/Locale-Maketext/lib/Locale/Maketext/TPJ13.pod. To my knowledge, this was the first published concept of translation scripting.

Games are especially susceptible to combinatorial effects of argument insertion. One such game is OpenTTD, and its developers resorted to expanding a simple ID-based translation system with facilities to handle plural, case and gender aspects of argument insertion. A document describing this can be found at http://wiki.openttd.org/OpenTTDDevBlackBook/Format_of_langfiles.

Axel Hecht, of the Mozilla Foundation, initiated in 2007 a new translation system, called L20n. It uses a cascading dictionary-like selection system for grammar manipulations, has some arithmetic and logical expressions capability, and defines a custom translation syntax and file format. While L20n comes from the Web oriented background, in principle it can be used in other contexts too. The current state of L20n, as well as an illustrated guide by way of examples, can be seen at http://l20n.org.