Word Frequencies and Lexical Diversity

You now know how to turn a paragraph into a clean list of tokens. The simplest — and most useful — thing you can do with that list is count. How many words are there? How many distinct words? Which words appear most? How varied is the vocabulary? These basic statistics are the foundation of search ranking, keyword extraction, spam scoring, and a surprising amount of "text analytics".

Counting words with FreqDist

You could count words with a plain collections.Counter, and that is exactly what is happening under the hood. NLTK wraps it in a FreqDist ("frequency distribution") that adds text-specific conveniences.

A few FreqDist methods you will use constantly:

• fd.most_common(n) — the n highest-frequency words as (word, count) pairs, already sorted. The everyday workhorse.
• fd.N() — the total number of word tokens (counting repeats).
• len(fd) — the number of distinct words (the vocabulary size).
• fd["word"] — the count of a word; missing words return 0 rather than raising, because FreqDist is a dict subclass.
• fd.freq("word") — the relative frequency, i.e. count divided by N().

Lexical diversity: how varied is the vocabulary?

Two texts can have the same length but very different richness. A text that says "very very very good" uses four words but only two distinct ones. The ratio of distinct words to total words is called lexical diversity, and it is a quick, classic measure of how repetitive or varied a text is.

$\text{lexical diversity} = \frac{\text{number of distinct words}}{\text{total number of words}}$

A value near 1 means almost every word is unique (highly varied); a value near 0 means heavy repetition.

The repetitive text scores low; the varied text scores near 1. Lexical diversity is used to compare writing styles, gauge reading level, flag machine-generated or spammy text (which often repeats), and track how an author's vocabulary changes across works.

The long tail: language is lopsided

If you sort words by frequency, you find a striking, universal pattern: a tiny number of words are extraordinarily common, and a huge number appear just once or twice. The single most frequent word ("the") often accounts for ~7% of all tokens by itself, and the "long tail" of rare words stretches on forever. (This regularity is known as Zipf's law.)

Words that appear exactly once have their own name — hapaxes — and FreqDist will list them for you. Let us visualize the lopsidedness with a bar chart of the top words.

The bar for "the" dwarfs the others, and a pile of words appear only once. This shape repeats in essentially every natural-language text.

A catch: raw frequency is dominated by stopwords

The long-tail shape leads directly to a practical problem. If you just ask "what are the most frequent words?", the answer is almost always the stopwords — "the", "a", "of" — which tell you nothing about the topic. The content words you actually care about are buried below them.

With stopwords, the top words are "the", "a", "that" — useless for guessing the topic. Without them, "text", "words", "search", "user", and "engine" rise to the top — and you can immediately tell this passage is about search. This is why keyword extraction almost always removes stopwords first, and it is a concrete payoff of the choice you studied two pages ago.

Your turn: basic text statistics

Check your understanding

You can now measure which words appear and how often. But counts treat every word as an isolated atom — they have no idea that "book" can be a noun or a verb. To capture that, we need to know each word's role in the sentence: part-of-speech tagging, next.