Measuring Lexical Style and Competence: The Type-Token Vocabulary Curve
Abstract
A personal computer is used to analyze samples from literary texts by thirteen
different authors, including passages from Genesis, Hemingway, Longfellow, Jane
Austen, Henry James, George Eliot, James Joyce, and Basic English (created by C. K.
Ogden). The total number of words (tokens) and the number of distinct vocabulary
words (types) are computed for each sample. The number of types are then plotted
against the number of tokens for eight of the texts. From these type-token curves,
inferences are drawn about both lexical style (vocabulary use) and lexical competence
(vocabulary size). For example, the curves for "Big Two-Hearted River" and for a
summary of Macbeth in Basic English nearly coincide for their first 1100 tokens, after
which they gradually diverge. This graphical pattern corresponds with intuitions that
Hemingway's prose reads much like Basic English but that it draws upon a larger total
vocabulary. The curve for Joyce's Ulysses, by contrast, rises much more rapidly than
that for a late passage from A Portrait of the Artist as a Young Man; however, after 800
tokens, the two curves begin to converge. This suggests that the difference between
Ulysses and Portrait is largely one of lexical style rather than competence. The highest type-token curve for the samples tested was that for Finnegans Wake; the lowest curve was for Genesis. Comparison with type-token statistics gathered by Kucera and Francis suggests that the curves for the Wake and Genesis are near the maxima and minima for English literature.
Part of
Citation
Style, 24 (Winter 1990): 584-599.
Rights
OpenAccess.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 License.