Here is a sample of German vocabulary, with the stemmed forms that will be generated by this algorithm:
| word | stem | word | stem | |||||||||
|
aufeinander aufeinanderbiss aufeinanderfolge aufeinanderfolgen aufeinanderfolgend aufeinanderfolgende aufeinanderfolgenden aufeinanderfolgender aufeinanderfolgt aufeinanderfolgten aufeinanderschlügen aufenthalt aufenthalten aufenthaltes auferlegen auferlegt auferlegten auferstand auferstanden auferstehen aufersteht auferstehung auferstünde auferwecken auferweckt auferzogen aufessen auffa auffallen auffallend auffallenden auffallender auffällig auffälligen auffälliges auffassen auffasst auffaßt auffassung auffassungsvermögen |
⇒ |
aufeinand aufeinanderbiss aufeinanderfolg aufeinanderfolg aufeinanderfolg aufeinanderfolg aufeinanderfolg aufeinanderfolg aufeinanderfolgt aufeinanderfolgt aufeinanderschlug aufenthalt aufenthalt aufenthalt auferleg auferlegt auferlegt auferstand auferstand aufersteh aufersteht aufersteh auferstund auferweck auferweckt auferzog aufess auffa auffall auffall auffall auffall auffall auffall auffall auffass auffasst auffasst auffass auffassungsvermog |
kategorie kategorien kategorisch kategorische kategorischen kategorischer kater katerliede katern katers käthchen kathedrale kathinka katholik katholische katholischen katholischer kattun kattunhalstücher katz kätzchen kätzchens katze katzen katzenschmer katzensprung katzenwürde kätzin kätzlein katzmann kauen kauerte kauf kaufe kaufen käufer kauffahrer kaufherr kaufleute käuflich |
⇒ |
kategori kategori kategor kategor kategor kategor kat katerlied kat kat kathch kathedral kathinka kathol kathol kathol kathol kattun kattunhalstuch katz katzch katzch katz katz katzenschm katzenspr katzenwurd katzin katzlein katzmann kau kauert kauf kauf kauf kauf kauffahr kaufherr kaufleut kauflich |
Despite its inflexional complexities, German has quite a simple suffix structure, so that, if one ignores the almost intractable problems of compound words, separable verb prefixes, and prefixed and infixed ge, an algorithmic stemmer can be made quite short. (Infixed zu can be removed algorithmically, but this minor feature is not shown here.) The umlaut in German is a regular feature of plural formation, so its removal is a natural feature of stemming, but this leads to certain false conflations (for example, schön, beautiful; schon, already).
There are a few short suffixes (for example, -t, -et) where removal would causes problems so the stemmer leaves these alone - for example holen, hole, hol and holest are stemmed to hol; ideally holt would be too, but a rule to remove -t would adversely affect too many other words.
For similar reasons, some suffixes are only removed when in R2: for example -end.
Famously, German allows for the formation of long compound words, written without spaces. For retrieval purposes, it is useful to be able to search on the parts of such words, as well as the on the complete words themselves. This is not just peculiar to German: Dutch, Danish, Norwegian, Swedish, Icelandic and Finnish have the same property. To split up compound words cannot be done without a dictionary, and the purely algorithmic stemmers presented here do not attempt it.
We would suggest, however, that the need for compound word splitting in these languages has been somewhat overstated. In the case of German:
There are many English compounds one would see no advantage in splitting,
| blackberry | blackboard | rainbow | coastguard | .... |
Many German compounds are like this,
| Bleistift (pencil) | = | Blei (lead) + Stift (stick) | ||
| Eisenbahn (railway) | = | Eisen (iron) + Bahn (road) | ||
| Unterseeboot (submarine) | = | under + sea + boat |
Other compounds correspond to what in English one would want to do by phrase searching, so they are ready made for that purpose,
| Gesundheitspflege | = | ‘health care’ | ||
| Fachhochschule | = | ‘technical college’ | ||
| Kunstmuseum | = | ‘museum of fine art’ |
In any case, longer compounds, especially involving personal names, are frequently hyphenated,
It is possible to construct participial adjectives of almost any length, but they are little used in contemporary German, and regarded now as poor style.
German includes the following accented forms,
and a special letter, ß, equivalent to double s.
The following letters are vowels:
First put u and y between vowels into upper case, and then do the following mappings,
(The rules here for ae, oe and ue were added in Snowball 2.3.0, but were previously present as a variant of the algorithm termed "german2"). The condition on the replacement of ue prevents the unwanted changing of quelle. Also note that feuer is not modified because the first part of the rule changes it to feUer, so ue is not found.)
R1 and R2 are first set up in the standard way (see the note on R1 and R2), but then R1 is adjusted so that the region before it contains at least 3 letters.
Define a valid s-ending as one of b, d, f, g, h, k, l, m, n, r or t.
Define a valid st-ending as the same list, excluding letter r.
Do each of steps 1, 2 and 3.
Step 1:
and if in R1 then delete (for (a) to (e)) or replace with l (for (f)). (Note that only the suffix needs to be in R1, the letter of the valid s-ending is not required to be.)
If an ending of group (c) is deleted, and the ending is preceded by niss, delete the final s.
(For example, äckern → äck, ackers → acker, armes → arm, bedürfnissen → bedürfnis)
Search for the longest among the following suffixes,
and delete if in R1.
(For example, derbsten → derbst by step 1, and derbst → derb by step 2, since b is a valid st-ending, and is preceded by just 3 letters)
Search for the longest among the following suffixes, and perform the action indicated.
Finally,
/*
Extra rule for -nisse ending added 11 Dec 2009
*/
routines (
prelude postlude
mark_regions
R1 R2
standard_suffix
)
externals ( stem )
integers ( p1 p2 x )
groupings ( v s_ending st_ending )
stringescapes {}
/* special characters */
stringdef a" '{U+00E4}'
stringdef o" '{U+00F6}'
stringdef u" '{U+00FC}'
stringdef ss '{U+00DF}'
define v 'aeiouy{a"}{o"}{u"}'
define s_ending 'bdfghklmnrt'
define st_ending s_ending - 'r'
define prelude as (
test repeat goto (
v [('u'] v <- 'U') or
('y'] v <- 'Y')
)
repeat (
[substring] among(
'{ss}' (<- 'ss')
'ae' (<- '{a"}')
'oe' (<- '{o"}')
'ue' (<- '{u"}')
'qu' ()
'' (next)
)
)
)
define mark_regions as (
$p1 = limit
$p2 = limit
test(hop 3 setmark x)
gopast v gopast non-v setmark p1
try($p1 < x $p1 = x) // at least 3
gopast v gopast non-v setmark p2
)
define postlude as repeat (
[substring] among(
'Y' (<- 'y')
'U' (<- 'u')
'{a"}' (<- 'a')
'{o"}' (<- 'o')
'{u"}' (<- 'u')
'' (next)
)
)
backwardmode (
define R1 as $p1 <= cursor
define R2 as $p2 <= cursor
define standard_suffix as (
do (
[substring] R1 among(
'em'
( not 'syst' // don't remove -em from words ending -system
delete
)
'ern' 'er'
'erin' 'erinnen' // conflate female versions of nouns
( delete
)
'e' 'en' 'es'
( delete
try (['s'] 'nis' delete)
)
's'
( s_ending delete
)
'ln' 'lns'
( <- 'l'
)
)
)
do (
[substring] R1 among(
'en' 'er' 'est'
( delete
)
'st'
( st_ending hop 3 delete
)
)
)
do (
[substring] R2 among(
'end' 'ung'
( delete
try (['ig'] not 'e' R2 delete)
)
'ig' 'ik' 'isch'
( not 'e' delete
)
'lich' 'heit'
( delete
try (
['er' or 'en'] R1 delete
)
)
'keit'
( delete
try (
[substring] R2 among(
'lich' 'ig'
( delete
)
)
)
)
)
)
)
)
define stem as (
do prelude
do mark_regions
backwards
do standard_suffix
do postlude
)