Why are a few words so common and most so rare?
A few words turn up constantly, while many appear only occasionally. Rank the words in a collection by frequency and an approximate pattern called Zipf’s law often emerges. The pattern is a description to investigate, not a guarantee that each rank has an exact fraction of the previous count or an explanation of why language works that way.
Questions you could explore
- Would treating “run,” “runs,” and “running” as one word change the graph?
- Where does the chosen collection depart from the fitted pattern?
Start reading
- Manning, Raghavan, and Schütze / Introduction to Information RetrievalZipf’s law: Modeling the distribution of terms
Defines the rank-frequency pattern and compares its fitted curve with words in a particular text collection.
Ready to try your ideas aloud?
Your topic carries over. Record up to 2½ minutes, depending on your plan.