DataslopeA Rule-Based Sentiment Classifier
The capstone — assemble tokenization, normalization, deliberate stopword choices, and negation handling into a working lexicon-based sentiment classifier. Watch why keeping negation matters, see how text flows through the whole pipeline, and learn the limits of rule-based methods.
Time to put it all together. In this final build we will write a rule-based sentiment classifier — a program that reads a sentence and decides whether it is positive, negative, or neutral — using nothing but the classic techniques from this course and a hand-made word list. No machine learning, no training data, no black box. Every decision it makes, you will be able to explain.
This capstone matters for two reasons. First, lexicon-based sentiment is genuinely used in the real world (it is transparent, needs no training data, and is easy to audit). Second, building it forces every earlier lesson to pay off at once — especially the warning about negation and stopwords.
The idea: score words against a lexicon
A sentiment lexicon is just two sets of words: ones that signal positive feeling ("good", "love", "excellent") and ones that signal negative feeling ("bad", "hate", "terrible"). To score a sentence, tokenize it, look up each word, add up the polarities, and read off the sign of the total. Here is the full flow — trace it top to bottom, because it is the entire course in one diagram.
Notice step B already encodes a decision you learned to make: we keep stopwords and negation, because — as the stopwords page warned — stripping "not" would flip a review's meaning. The pipeline you are building is a direct application of the judgment from the "choosing your pipeline" page.
A naive first attempt (and why it fails)
Let us start with the obvious version: count positive words, count negative words, compare. It will work on simple sentences and then fail in a way you can now predict.
The first two are right. The third — "It is not good" — is scored positive, because the naive counter sees the word "good" and is blind to the "not" sitting right in front of it. This is the exact negation failure we have been warning about since the stopwords page, now caught red-handed.
The naive bag-of-words sentiment trap
Counting positive and negative words independently is a bag-of-words model, and it inherits bag-of-words' blindness to order. "not good" contains the positive word "good", so the naive model calls it positive. Fixing this is exactly why we kept "not" in our tokens and why local order (the previous token) has to be part of the logic.
Adding negation handling
The fix is to track whether the previous token was a negation and, if so, flip the polarity of the next sentiment word. We also reset the negation at punctuation, so it does not bleed across clauses. This is a tiny amount of "local order" awareness — the same insight behind n-grams.
Now "It is not good" is correctly negative, and — satisfyingly — "This is not bad at all" comes out positive, because the negation flipped the negative word "bad" into a positive contribution. You just taught a program the difference between "good" and "not good" using only tokenization and a single boolean flag for local order. That is classic NLP working exactly as designed.
Every lesson showed up here
Look back at what this classifier used: tokenization (to get words and split punctuation), case folding (so "Good" matches "good"), a deliberate choice to keep stopwords (so "not" survives), lexicon lookup (a form of feature extraction), and local-order awareness (the n-gram insight, as a negation flag). The whole course converged on one small, explainable program.
The limits of rules — and when to graduate to learning
Rule-based sentiment is transparent and needs no training data, but it has real ceilings you should respect:
- Sarcasm and irony. "Oh great, another delay." The words say positive; the meaning is negative. Rules cannot see the wink.
- Context and domain. "unpredictable" is bad for a car's brakes but good for a thriller's plot. A fixed lexicon cannot know which.
- Coverage. Any word missing from your lexicon contributes nothing. Slang, typos, and new expressions slip through.
- Intensity and mixed opinion. "good" and "phenomenal" both score +1; a review that is half-praise, half-complaint nets out to a muddle.
These limits are why machine-learning approaches exist: instead of a hand-written lexicon, they learn word sentiment (and more) from thousands of labeled examples, picking up sarcasm cues, domain meanings, and intensity that no hand-built list captures. But notice — those models still tokenize, normalize, and make the very preprocessing choices you practiced here. You have built the foundation they stand on.
Why start with rules?
Even when you eventually reach for machine learning, a rule-based baseline is worth building first: it is fast, transparent, needs no labeled data, and gives you a score to beat. If a simple lexicon already solves your problem, you may not need anything more complicated — recall the very first page's advice that a clear rule often beats a model.
The naive word-counting classifier labels "It is not good" as positive. Which earlier lesson explains this failure most directly?
Stemming should have been applied first
Counting positive/negative words independently ignores word order, so "not good" looks positive because of "good" — the same order-blindness as bag-of-words, fixable with negation/local-order handling
The text was not lowercased
"good" is a stopword and should have been removed
Your turn: build the negation-aware classifier
This is the capstone challenge. Implement the negation-aware scoring loop using the lexicon and helper sets provided for you.
Write classify(text) that returns "positive", "negative", or
"neutral". Use the provided POSITIVE, NEGATIVE, NEGATIONS, and
RESET sets.
Algorithm:
- Tokenize
textand lowercase every token. - Keep a running
score(start at 0) and anegateflag (startFalse). - For each token, in order:
- If it is in
NEGATIONS, setnegate = Trueand move on. - If it is in
RESET(punctuation or "but"), setnegate = Falseand move on. - Otherwise compute its polarity:
+1if inPOSITIVE,-1if inNEGATIVE, else0. If the polarity is nonzero andnegateisTrue, flip the polarity and resetnegatetoFalse. Add the polarity toscore.
- Return
"positive"ifscore > 0,"negative"ifscore < 0, otherwise"neutral".
This should make "this is not good" negative and "this is not bad" positive.
Check your understanding
What is the core idea of a lexicon-based (rule-based) sentiment classifier?
It learns word sentiment from thousands of labeled training examples
It scores text by looking each word up in hand-made lists of positive and negative words and summing the polarities
It translates the text and counts the characters
It uses a neural network to predict sentiment
Why does the classifier deliberately keep stopwords like "not" instead of removing them?
Because removing stopwords is computationally expensive
Because "not" is a negation that reverses sentiment; removing it (it is on the standard stopword list) would turn "not good" into "good" and flip the result
Because stopwords are the most positive words
Because NLTK forbids removing stopwords
Which of these is a genuine limitation of rule-based sentiment that motivates machine-learning approaches?
It is unable to lowercase text
It cannot detect sarcasm ("Oh great, another delay"), handle domain-specific meanings, or score words missing from its lexicon
It runs only on Tuesdays
It cannot process more than one sentence
In the negation-aware algorithm, what is the purpose of the RESET set
(punctuation and "but")?
To delete those tokens from the text
To clear the negation flag at clause boundaries so a negation does not incorrectly carry over and flip a word in a later clause
To add extra weight to positive words
To convert the text to uppercase
Even when you plan to use machine learning later, why is building a rule-based baseline first a good idea?
It is required before any model can be trained
It is fast, transparent, needs no labeled data, and gives a concrete score to beat — and sometimes it already solves the problem well enough
It guarantees higher accuracy than any model
It removes the need to tokenize text
You have built a complete, explainable NLP program from raw text to a labeled verdict — using only the classic toolkit. That is a real accomplishment. The final page points you toward where to go from here.
From Text to Features: Bag-of-Words
Algorithms need numbers, not words. The bag-of-words model turns each document into a vector of word counts over a shared vocabulary. Building a document-term matrix by hand, binary vs. count features, and the limitations (lost order, sparsity, no semantics) that motivate n-grams and beyond.
Where to Go Next
A recap of the classic NLP pipeline you now command, and a map of where to go from here — machine-learning text classification, modern toolkits, embeddings, and transformers — all built on the foundations you just learned.