Danish stemming algorithm

Links to resources

• Scandinavian language stemmers
• Javascript demo
• The stemmer in Snowball
• Sample Danish vocabulary
• Its stemmed equivalent
• Danish stop word list (UTF-8 encoding)
• Danish language on Wikipedia

Here is a sample of Danish vocabulary, with the stemmed forms that will be generated by this algorithm:

The stemming algorithm

The Danish alphabet includes the following additional letters,

æ   å   ø

The following letters are vowels:

a   e   i   o   u   y   æ   å   ø

R2 is not used. R1 is set up by the following steps:

1. If the word contains an apostrophe, R1 is set to start after the first apostrophe.
2. Otherwise, R1 is set in the standard way (see note).
3. In either case, R1 is then adjusted so that the region before it contains at least 3 characters.

Define a valid s-ending as one of

a   b   c   d   f   g   h   j   k   l   m   n   o   p   r   t   v   y   z   å

Do each of steps 1, 2, 3, 4 and 5.

Step 1:

Search for the longest among the following suffixes in R1, and perform the action indicated.

(a) hed   ethed   ered   e   erede   ende   erende   ene   erne   ere   en   heden   eren   er   heder   erer   heds   es   endes   erendes   enes   ernes   eres   ens   hedens   erens   ers   ets   erets   et   eret

delete

(b) s

delete if preceded by a valid s-ending

(Note that only the suffix needs to be in R1, the letter of the valid s-ending is not required to be.)

Step 2:

Search for one of the following suffixes in R1, and if found delete the last letter.

gd   dt   gt   kt

(For example, friskt → frisk)

If the word ends igst, remove the final st.

Search for the longest among the following suffixes in R1, and perform the action indicated.

(a) ig   lig   elig   els

delete, and then repeat step 2

(b) løst

replace with løs

If the word ends with one of bb dd ff gg kk ll mm nn pp rr ss tt in R1, then remove the last letter.

(For example, bestemmelse → bestemmels (step 1) → bestemm (step 3a) → bestem in this step.)

If the word ends with an apostrophe (') then remove it.

(For example, cd'en → cd' (step 1) → cd in this step.)

Design Notes

Apostrophe is used in Danish to separate Danish suffixes from words of foreign origin. This is commonly seen with acronyms/initialisms, some of which are only two characters and don't contain vowels (e.g. cd'en) so the definition of R1 includes a special case for apostrophe.

History of functional changes to the algorithm

• Snowball 2.0.0 (2018-11-15): Tightened definition of "consonant" to be "a character from ASCII a-z which isn't a vowel" (originally this was "A consonant is defined as a non-vowel" but that caused mangling of alphanumeric codes).
• Snowball 3.1.0: Restrict undoubling to a subset of consonants which actually occur in this situation in Danish words (this change didn't affect the stemming of any words in our sample vocabulary).
• Snowball 3.1.0: Add special handling for apostrophe.

The same algorithm in Snowball

routines (
           mark_regions
           main_suffix
           consonant_pair
           other_suffix
           undouble
)

externals ( stem )

strings ( ch )

integers ( p1 x )

groupings ( c v s_ending )

stringescapes {}

/* special characters */

stringdef ae   '{U+00E6}'
stringdef ao   '{U+00E5}'
stringdef o/   '{U+00F8}'

define c 'bcdfghjklmnpqrstvwxz'

define v 'aeiouy{ae}{ao}{o/}'

define s_ending  'abcdfghjklmnoprtvyz{ao}'

define mark_regions as (

    $p1 = limit

    test ( hop 3 setmark x )
    gopast v  gopast non-v  setmark p1
    try ( $p1 < x  $p1 = x )
)

backwardmode (

    define main_suffix as (
        setlimit tomark p1 for ([substring])
        among(

            'hed' 'ethed' 'ered' 'e' 'erede' 'ende' 'erende' 'ene' 'erne' 'ere'
            'en' 'heden' 'eren' 'er' 'heder' 'erer' 'heds' 'es' 'endes'
            'erendes' 'enes' 'ernes' 'eres' 'ens' 'hedens' 'erens' 'ers' 'ets'
            'erets' 'et' 'eret'
                (delete)
            's'
                (s_ending delete)
        )
    )

    define consonant_pair as (
        test (
            setlimit tomark p1 for ([substring])
            among(
                'gd' // significant in the call from other_suffix
                'dt' 'gt' 'kt'
            )
        )
        next] delete
    )

    define other_suffix as (
        do ( ['st'] 'ig' delete )
        setlimit tomark p1 for ([substring])
        among(
            'ig' 'lig' 'elig' 'els'
                (delete do consonant_pair)
            'l{o/}st'
                (<-'l{o/}s')
        )
    )
    define undouble as (
        setlimit tomark p1 for ([c] ->ch)
        ch
        delete
    )
)

define stem as (

    do mark_regions
    backwards (
        do main_suffix
        do consonant_pair
        do other_suffix
        do undouble
    )
)