The walk example doesn't quite hold up.

If you put:

    test walk walker walking walked

into the tokenizer you will see the following tokens:

    [test][ walk][ walk][er][ walking][ walked]

Only walker is broken up into two different tokens.

I added "test" to that because walk at the start doesn't include the leading space and [walk] and [ walk] are different tokens.

For even more fun, [walker] is a distinct token if it doesn't include the leading space.

    test walker floorwalker foowalker

becomes:

    [test][ walk][er][ floor][walker][ fo][ow][alker]

How we think of words doesn't cleanly map to tokens.

(Late edit)

    walker floorwalker

becomes tokenized as:

    [walker][ floor][walker]

So in that case, they're the same token. It's curious how white space influences the word to token making.