Merge pull request #3785 from mgavioli/new_i18n_API

Separate I18N calls for immutable and for pluraliseable strings. Fixes #3687

core/text/i18n/doc.odin

@@ -1,31 +1,44 @@
 
 /*
-The `i18n` package is flexible and easy to use.
+The `i18n` package is a flexible and easy to use way to localise applications.
 
-It has one call to get a translation: `get`, which the user can alias into something like `T`.
+It has two calls to get a translation: `get()` and `get_n()`, which the user can alias into something like `T` and `Tn`
+with statements like:
+	T  :: i18n.get
+	Tn :: i18n.get_n.
 
-`get`, referred to as `T` here, has a few different signatures.
+`get()` is used for retrieving the translation of sentences which **never** change in form,
-All of them will return the key if the entry can't be found in the active translation catalog.
+like for instance "Connection established" or "All temporary files have been deleted".
+Note that the number (singular, dual, plural, whatever else) is not relevant: the sentence is fixed and it will have only one possible translation in any other language.
 
-- `T(key)`              returns the translation of `key`.
+`get_n()` is used for retrieving the translations of sentences which change according to the number of items referenced.
-- `T(key, n)`           returns a pluralized translation of `key` according to value `n`.
+The various signatures of `get_n()` have one more parameter, `n`, which will receive that number and be used
+to select the correct form according to the pluralizer attached to the message catalogue when initially loaded;
+for instance, to summarise a rather complex matter, some languages use the singular form when referring to 0 items and some use the (only in their case) plural forms;
+also, languages may have more or less quantifier forms than a single singular form and a universal plural form:
+for instance, Chinese has just one form for any quantity, while Welsh may have up to 6 different forms for specific different quantities.
 
-- `T(section, key)`     returns the translation of `key` in `section`.
+Both `get()` and `get_n()`, referred to as `T` and `Tn` here, have several different signatures.
-- `T(section, key, n)`  returns a pluralized translation of `key` in `section` according to value `n`.
+All of them will return the key if the entry can't be found in the active translation catalogue.
+By default lookup take place in the global `i18n.ACTIVE` catalogue for ease of use, unless a specific catalogue is supplied.
 
-By default lookup take place in the global `i18n.ACTIVE` catalog for ease of use.
+- `T(key)`                   returns the translation of `key`.
-If you want to override which translation to use, for example in a language preview dialog, you can use the following:
+- `T(key, catalog)`          returns the translation of `key` from explictly supplied catalogue.
+- `T(section, key)`          returns the translation of `key` in `section`.
+- `T(section, key, catalog)` returns the translation of `key` in `section` from explictly supplied catalogue.
 
-- `T(key, n, catalog)`           returns the pluralized version of `key` from explictly supplied catalog.
+- `Tn(key, n)`                   returns the translation of `key` according to number of items `n`.
-- `T(section, key, n, catalog)`  returns the pluralized version of `key` in `section` from explictly supplied catalog.
+- `Tn(key, n, catalog)`          returns the translation of `key` from explictly supplied catalogue.
+- `Tn(section, key, n)`          returns the translation of `key` in `section` according to number of items `n`.
+- `Tn(section, key, n, catalog)` returns the translation of `key` in `section` according to number of items `n` from explictly supplied catalogue.
 
 If a catalog has translation contexts or sections, then omitting it in the above calls looks up in section "".
 
-The default pluralization rule is n != 1, which is to say that passing n == 1 (or not passing n) returns the singular form.
+The default pluralization rule is `n != 1`, which is to say that passing `n == 1` returns the singular form (in slot 0).
-Passing n != 1 returns plural form 1.
+Passing `n != 1` returns the plural form in slot 1 (if any).
 
 Should a language not conform to this rule, you can pass a pluralizer procedure to the catalog parser.
-This is a procedure that maps an integer to an integer, taking a value and returning which plural slot should be used.
+This is a procedure that maps an integer to an integer, taking a quantity and returning which plural slot should be used.
 
 You can also assign it to a loaded catalog after parsing, of course.
 
@@ -34,24 +47,21 @@ Example:
 	import "core:fmt"
 	import "core:text/i18n"
 
-	T :: i18n.get
+	T  :: i18n.get
+	Tn :: i18n.get_n
 
 	mo :: proc() {
 		using fmt
 
 		err: i18n.Error
 
-		/*
+		// Parse MO file and set it as the active translation so we can omit `get`'s "catalog" parameter.
-			Parse MO file and set it as the active translation so we can omit `get`'s "catalog" parameter.
-		*/
 		i18n.ACTIVE, err = i18n.parse_mo(#load("translations/nl_NL.mo"))
 		defer i18n.destroy()
 
 		if err != .None { return }
 
-		/*
+		// These are in the .MO catalog.
-			These are in the .MO catalog.
-		*/
 		println("-----")
 		println(T(""))
 		println("-----")
@@ -60,13 +70,11 @@ Example:
 		println(T("Hellope, World!"))
 		println("-----")
 		// We pass 1 into `T` to get the singular format string, then 1 again into printf.
-		printf(T("There is %d leaf.\n", 1), 1)
+		printf(Tn("There is %d leaf.\n", 1), 1)
 		// We pass 42 into `T` to get the plural format string, then 42 again into printf.
-		printf(T("There is %d leaf.\n", 42), 42)
+		printf(Tn("There is %d leaf.\n", 42), 42)
 
-		/*
+		// This isn't in the translation catalog, so the key is passed back untranslated.
-			This isn't in the translation catalog, so the key is passed back untranslated.
-		*/
 		println("-----")
 		println(T("Come visit us on Discord!"))
 	}
@@ -76,19 +84,13 @@ Example:
 
 		err: i18n.Error
 
-		/*
+		// Parse QT file and set it as the active translation so we can omit `get`'s "catalog" parameter.
-			Parse QT file and set it as the active translation so we can omit `get`'s "catalog" parameter.
-		*/
 		i18n.ACTIVE, err = i18n.parse_qt(#load("translations/nl_NL-qt-ts.ts"))
 		defer i18n.destroy()
 
-		if err != .None {
+		if err != .None { return }
-			return
-		}
 
-		/*
+		// These are in the .TS catalog. As you can see they have sections.
-			These are in the .TS catalog. As you can see they have sections.
-		*/
 		println("--- Page section ---")
 		println("Page:Text for translation =", T("Page", "Text for translation"))
 		println("-----")
@@ -99,8 +101,8 @@ Example:
 		println("-----")
 		println("--- apple_count section ---")
 		println("apple_count:%d apple(s) =")
-		println("\t 1  =", T("apple_count", "%d apple(s)", 1))
+		println("\t 1  =", Tn("apple_count", "%d apple(s)", 1))
-		println("\t 42 =", T("apple_count", "%d apple(s)", 42))
+		println("\t 42 =", Tn("apple_count", "%d apple(s)", 42))
 	}
 */
 package i18n

core/text/i18n/i18n.odin

@@ -10,23 +10,13 @@ package i18n
 */
 import "core:strings"
 
-/*
+// Currently active catalog.
-	TODO:
-	- Support for more translation catalog file formats.
-*/
-
-/*
-	Currently active catalog.
-*/
 ACTIVE: ^Translation
 
 // Allow between 1 and 255 plural forms. Default: 10.
 MAX_PLURALS :: min(max(#config(ODIN_i18N_MAX_PLURAL_FORMS, 10), 1), 255)
 
-/*
+// The main data structure. This can be generated from various different file formats, as long as we have a parser for them.
-	The main data structure. This can be generated from various different file formats, as long as we have a parser for them.
-*/
-
 Section :: map[string][]string
 
 Translation :: struct {
@@ -37,34 +27,24 @@ Translation :: struct {
 }
 
 Error :: enum {
-	/*
+	// General return values.
-		General return values.
-	*/
 	None = 0,
 	Empty_Translation_Catalog,
 	Duplicate_Key,
 
-	/*
+	// Couldn't find, open or read file.
-		Couldn't find, open or read file.
-	*/
 	File_Error,
 
-	/*
+	// File too short.
-		File too short.
-	*/
 	Premature_EOF,
 
-	/*
+	// GNU Gettext *.MO file errors.
-		GNU Gettext *.MO file errors.
-	*/
 	MO_File_Invalid_Signature,
 	MO_File_Unsupported_Version,
 	MO_File_Invalid,
 	MO_File_Incorrect_Plural_Count,
 
-	/*
+	// Qt Linguist *.TS file errors.
-		Qt Linguist *.TS file errors.
-	*/
 	TS_File_Parse_Error,
 	TS_File_Expected_Context,
 	TS_File_Expected_Context_Name,
@@ -85,73 +65,142 @@ DEFAULT_PARSE_OPTIONS :: Parse_Options{
 }
 
 /*
-	Several ways to use:
+	Returns the first translation string for the passed `key`.
-	- get(key), which defaults to the singular form and i18n.ACTIVE catalog, or
+	It is also aliased with `get()`.
-	- get(key, number), which returns the appropriate plural from the active catalog, or
-	- get(key, number, catalog) to grab text from a specific one.
-*/
-get_single_section :: proc(key: string, number := 1, catalog: ^Translation = ACTIVE) -> (value: string) {
-	/*
-		A lot of languages use singular for 1 item and plural for 0 or more than 1 items. This is our default pluralize rule.
-	*/
-	plural := 1 if number != 1 else 0
 
-	if catalog.pluralize != nil {
+	Two ways to use it:
-		plural = catalog.pluralize(number)
+	- get(key), which defaults to the `i18n.ACTIVE` catalogue, or
-	}
+	- get(key, catalog) to grab text from a specific loaded catalogue
-	return get_by_slot(key, plural, catalog)
+
+	Inputs:
+	- key:     the string to translate
+	- catalog: the catalogue to use for the translation (defaults to i18n.ACTIVE)
+
+	Returns:   the translated string, or the original `key` if no translation was found.
+*/
+get_single_section :: proc(key: string, catalog: ^Translation = ACTIVE) -> (value: string) {
+	return get_by_slot(key, 0, catalog)
 }
 
 /*
-	Several ways to use:
+	Returns the first translation string for the passed `key` in a specific section or context.
-	- get(section, key), which defaults to the singular form and i18n.ACTIVE catalog, or
+	It is also aliases with `get()`.
-	- get(section, key, number), which returns the appropriate plural from the active catalog, or
-	- get(section, key, number, catalog) to grab text from a specific one.
-*/
-get_by_section :: proc(section, key: string, number := 1, catalog: ^Translation = ACTIVE) -> (value: string) {
-	/*
-		A lot of languages use singular for 1 item and plural for 0 or more than 1 items. This is our default pluralize rule.
-	*/
-	plural := 1 if number != 1 else 0
 
-	if catalog.pluralize != nil {
+	Two ways to use it:
-		plural = catalog.pluralize(number)
+	- get(section, key), which defaults to the `i18n.ACTIVE` catalogue, or
-	}
+	- get(section, key, catalog) to grab text from a specific loaded catalogue
-	return get_by_slot(section, key, plural, catalog)
+
+	Inputs:
+	- section: the catalogue section (sometimes also called 'context') in which to look up the translation
+	- key:     the string to translate
+	- catalog: the catalogue to use for the translation (defaults to i18n.ACTIVE)
+
+	Returns:   the translated string, or the original `key` if no translation was found.
+*/
+get_by_section :: proc(section, key: string, catalog: ^Translation = ACTIVE) -> (value: string) {
+	return get_by_slot(section, key, 0, catalog)
 }
+
 get :: proc{get_single_section, get_by_section}
 
 /*
-	Several ways to use:
+	Returns the translation string for the passed `key` in a specific plural form (if present in the catalogue).
-	- get_by_slot(key), which defaults to the singular form and i18n.ACTIVE catalog, or
+	It is also aliased with `get_n()`.
-	- get_by_slot(key, slot), which returns the requested plural from the active catalog, or
+
-	- get_by_slot(key, slot, catalog) to grab text from a specific one.
+	Two ways to use it:
+	- get_n(key, quantity), which returns the appropriate plural from the active catalogue, or
+	- get_n(key, quantity, catalog) to grab text from a specific loaded catalogue
+
+	Inputs:
+	- key:      the string to translate
+	- quantity: the quantity of item to be used to select the correct plural form
+	- catalog:  the catalogue to use for the translation (defaults to i18n.ACTIVE)
+
+	Returns:    the translated string, or the original `key` if no translation was found.
+*/
+get_single_section_with_quantity :: proc(key: string, quantity: int, catalog: ^Translation = ACTIVE) -> (value: string) {
+	/*
+		A lot of languages use singular for 1 item and plural for 0 or more than 1 items. This is our default pluralize rule.
+	*/
+	slot := 1 if quantity != 1 else 0
+
+	if catalog.pluralize != nil {
+		slot = catalog.pluralize(quantity)
+	}
+	return get_by_slot(key, slot, catalog)
+}
+
+/*
+	Returns the translation string for the passed `key` in a specific plural form (if present in the catalogue)
+	in a specific section or context.
+	It is also aliases with `get_n()`.
+
+	Two ways to use it:
+	- get(section, key, quantity), which returns the appropriate plural from the active catalogue, or
+	- get(section, key, quantity, catalog) to grab text from a specific loaded catalogue
+
+	Inputs:
+	- section: the catalogue section (sometime also called 'context') from which to lookup the translation
+	- key:     the string to translate
+	- qantity: the quantity of item to be used to select the correct plural form
+	- catalog: the catalogue to use for the translation (defaults to i18n.ACTIVE)
+
+	Returns:   the translated string, or the original `key` if no translation was found
+*/
+get_by_section_with_quantity :: proc(section, key: string, quantity: int, catalog: ^Translation = ACTIVE) -> (value: string) {
+	/*
+		A lot of languages use singular for 1 item and plural for 0 or more than 1 items. This is our default pluralize rule.
+	*/
+	slot := 1 if quantity != 1 else 0
+
+	if catalog.pluralize != nil {
+		slot = catalog.pluralize(quantity)
+	}
+	return get_by_slot(section, key, slot, catalog)
+}
+get_n :: proc{get_single_section_with_quantity, get_by_section_with_quantity}
+
+/*
+	Two ways to use:
+	- get_by_slot(key, slot), which returns the requested plural from the active catalogue, or
+	- get_by_slot(key, slot, catalog) to grab text from a specific loaded catalogue.
 
 	If a file format parser doesn't (yet) support plural slots, each of the slots will point at the same string.
+	- section: the catalogue section (sometime also called 'context') from which to lookup the translation
+
+	Inputs:
+	- key:     the string to translate.
+	- slot:    the translation slot to choose (slots refer to plural forms specific for each language and their meaning changes from catalogue to catalogue).
+	- catalog: the catalogue to use for the translation (defaults to i18n.ACTIVE)
+
+	Returns:   the translated string, or the original `key` if no translation was found.
 */
-get_by_slot_single_section :: proc(key: string, slot := 0, catalog: ^Translation = ACTIVE) -> (value: string) {
+get_by_slot_single_section :: proc(key: string, slot: int, catalog: ^Translation = ACTIVE) -> (value: string) {
 	return get_by_slot_by_section("", key, slot, catalog)
 }
 
 /*
-	Several ways to use:
+	Two ways to use:
-	- get_by_slot(key), which defaults to the singular form and i18n.ACTIVE catalog, or
 	- get_by_slot(key, slot), which returns the requested plural from the active catalog, or
 	- get_by_slot(key, slot, catalog) to grab text from a specific one.
 
 	If a file format parser doesn't (yet) support plural slots, each of the slots will point at the same string.
+
+	Inputs:
+	- section: the catalogue section (sometime also called 'context') from which to lookup the translation
+	- key:     the string to translate.
+	- slot:    the translation slot to choose (slots refer to plural forms specific for each language and their meaning changes from catalogue to catalogue).
+	- catalog: the catalogue to use for the translation (defaults to i18n.ACTIVE)
+
+	Returns:   the translated string or the original `key` if no translation was found.
 */
-get_by_slot_by_section :: proc(section, key: string, slot := 0, catalog: ^Translation = ACTIVE) -> (value: string) {
+get_by_slot_by_section :: proc(section, key: string, slot: int, catalog: ^Translation = ACTIVE) -> (value: string) {
 	if catalog == nil || section not_in catalog.k_v {
-		/*
+		// Return the key if the catalog catalog hasn't been initialized yet, or the section is not present.
-			Return the key if the catalog catalog hasn't been initialized yet, or the section is not present.
-		*/
 		return key
 	}
 
-	/*
+	// Return the translation from the requested slot if this key is known, else return the key.
-		Return the translation from the requested slot if this key is known, else return the key.
-	*/
 	if translations, ok := catalog.k_v[section][key]; ok {
 		plural := min(max(0, slot), len(catalog.k_v[section][key]) - 1)
 		return translations[plural]
@@ -161,7 +210,6 @@ get_by_slot_by_section :: proc(section, key: string, slot := 0, catalog: ^Transl
 get_by_slot :: proc{get_by_slot_single_section, get_by_slot_by_section}
 
 /*
-	Same for destroy:
 	- destroy(), to clean up the currently active catalog catalog i18n.ACTIVE
 	- destroy(catalog), to clean up a specific catalog.
 */

tests/core/text/i18n/test_core_text_i18n.odin

@@ -5,6 +5,7 @@ import "core:testing"
 import "core:text/i18n"
 
 T :: i18n.get
+Tn :: i18n.get_n
 
 Test :: struct {
 	section: string,
@@ -47,7 +48,8 @@ test_custom_pluralizer :: proc(t: ^testing.T) {
 			{"", "Message1/plural",               "This is message 1",             1},
 			{"", "Message1/plural",               "This is message 1 - plural A",  1_000_000},
 			{"", "Message1/plural",               "This is message 1 - plural B",  42},
-			// This isn't in the catalog, so should ruturn the key.
+
+			// This isn't in the catalog, so should return the key.
 			{"", "Come visit us on Discord!",     "Come visit us on Discord!",      1},
 		},
 	})
@@ -61,11 +63,11 @@ test_mixed_context :: proc(t: ^testing.T) {
 		plural = nil,
 		tests  = {
 			// These are in the catalog.
-			{"",        "Message1",               "This is message 1 without Context", 1},
+			{"",        "Message1",               "This is message 1 without Context",-1},
-			{"Context", "Message1",               "This is message 1 with Context",    1},
+			{"Context", "Message1",               "This is message 1 with Context",   -1},
 
 			// This isn't in the catalog, so should ruturn the key.
-			{"", "Come visit us on Discord!",     "Come visit us on Discord!",         1},
+			{"", "Come visit us on Discord!",     "Come visit us on Discord!",        -1},
 		},
 	})
 }
@@ -90,15 +92,15 @@ test_nl_mo :: proc(t: ^testing.T) {
 		plural = nil, // Default pluralizer
 		tests  = {
 			// These are in the catalog.
-			{"", "There are 69,105 leaves here.", "Er zijn hier 69.105 bladeren.",  1},
+			{"", "There are 69,105 leaves here.", "Er zijn hier 69.105 bladeren.", -1},
-			{"", "Hellope, World!",               "Hallo, Wereld!",                 1},
+			{"", "Hellope, World!",               "Hallo, Wereld!",                -1},
 			{"", "There is %d leaf.\n",           "Er is %d blad.\n",               1},
 			{"", "There are %d leaves.\n",        "Er is %d blad.\n",               1},
 			{"", "There is %d leaf.\n",           "Er zijn %d bladeren.\n",        42},
 			{"", "There are %d leaves.\n",        "Er zijn %d bladeren.\n",        42},
 
 			// This isn't in the catalog, so should ruturn the key.
-			{"", "Come visit us on Discord!",     "Come visit us on Discord!",      1},
+			{"", "Come visit us on Discord!",     "Come visit us on Discord!",     -1},
 		},
 	})
 }
@@ -111,15 +113,15 @@ test_qt_linguist :: proc(t: ^testing.T) {
 		plural = nil, // Default pluralizer
 		tests  = {
 			// These are in the catalog.
-			{"Page",          "Text for translation",           "Tekst om te vertalen",        1},
+			{"Page",          "Text for translation",           "Tekst om te vertalen",       -1},
-			{"Page",          "Also text to translate",         "Ook tekst om te vertalen",    1},
+			{"Page",          "Also text to translate",         "Ook tekst om te vertalen",   -1},
-			{"installscript", "99 bottles of beer on the wall", "99 flessen bier op de muur",  1},
+			{"installscript", "99 bottles of beer on the wall", "99 flessen bier op de muur", -1},
 			{"apple_count",   "%d apple(s)",                    "%d appel",                    1},
 			{"apple_count",   "%d apple(s)",                    "%d appels",                  42},
 
 			// These aren't in the catalog, so should ruturn the key.
-			{"",              "Come visit us on Discord!",      "Come visit us on Discord!",   1},
+			{"",              "Come visit us on Discord!",      "Come visit us on Discord!",  -1},
-			{"Fake_Section",  "Come visit us on Discord!",      "Come visit us on Discord!",   1},
+			{"Fake_Section",  "Come visit us on Discord!",      "Come visit us on Discord!",  -1},
 		},
 	})
 }
@@ -133,16 +135,16 @@ test_qt_linguist_merge_sections :: proc(t: ^testing.T) {
 		options = {merge_sections = true},
 		tests   = {
 			// All of them are now in section "", lookup with original section should return the key.
-			{"",              "Text for translation",           "Tekst om te vertalen",            1},
+			{"",              "Text for translation",           "Tekst om te vertalen",           -1},
-			{"",              "Also text to translate",         "Ook tekst om te vertalen",        1},
+			{"",              "Also text to translate",         "Ook tekst om te vertalen",       -1},
-			{"",              "99 bottles of beer on the wall", "99 flessen bier op de muur",      1},
+			{"",              "99 bottles of beer on the wall", "99 flessen bier op de muur",     -1},
 			{"",              "%d apple(s)",                    "%d appel",                        1},
 			{"",              "%d apple(s)",                    "%d appels",                      42},
 
 			// All of them are now in section "", lookup with original section should return the key.
-			{"Page",          "Text for translation",           "Text for translation",            1},
+			{"Page",          "Text for translation",           "Text for translation",           -1},
-			{"Page",          "Also text to translate",         "Also text to translate",          1},
+			{"Page",          "Also text to translate",         "Also text to translate",         -1},
-			{"installscript", "99 bottles of beer on the wall", "99 bottles of beer on the wall",  1},
+			{"installscript", "99 bottles of beer on the wall", "99 bottles of beer on the wall", -1},
 			{"apple_count",   "%d apple(s)",                    "%d apple(s)",                     1},
 			{"apple_count",   "%d apple(s)",                    "%d apple(s)",                    42},
 		},
@@ -175,7 +177,7 @@ test :: proc(t: ^testing.T, suite: Test_Suite, loc := #caller_location) {
 
 	if err == .None {
 		for test in suite.tests {
-			val := T(test.section, test.key, test.n, cat)
+			val := test.n > -1 ? Tn(test.section, test.key, test.n, cat): T(test.section, test.key, cat)
 			testing.expectf(t, val == test.val, "Expected key `%v` from section `%v`'s form for value `%v` to equal `%v`, got `%v`", test.key, test.section, test.n, test.val, val, loc=loc)
 		}
 	}