Measuring Lexical Style and Competence: The Type-Token Vocabulary Curve

Abstract

A personal computer is used to analyze samples from literary texts by thirteen different authors, including passages from Genesis, Hemingway, Longfellow, Jane Austen, Henry James, George Eliot, James Joyce, and Basic English (created by C. K. Ogden). The total number of words (tokens) and the number of distinct vocabulary words (types) are computed for each sample. The number of types are then plotted against the number of tokens for eight of the texts. From these type-token curves, inferences are drawn about both lexical style (vocabulary use) and lexical competence (vocabulary size). For example, the curves for "Big Two-Hearted River" and for a summary of Macbeth in Basic English nearly coincide for their first 1100 tokens, after which they gradually diverge. This graphical pattern corresponds with intuitions that Hemingway's prose reads much like Basic English but that it draws upon a larger total vocabulary. The curve for Joyce's Ulysses, by contrast, rises much more rapidly than that for a late passage from A Portrait of the Artist as a Young Man; however, after 800 tokens, the two curves begin to converge. This suggests that the difference between Ulysses and Portrait is largely one of lexical style rather than competence. The highest type-token curve for the samples tested was that for Finnegans Wake; the lowest curve was for Genesis. Comparison with type-token statistics gathered by Kucera and Francis suggests that the curves for the Wake and Genesis are near the maxima and minima for English literature.