12 minute read

Introduction to Naive Bayes

Imagine you’re trying to guess if an email is spam. You might look for certain words like “lottery” or “free.” Naive Bayes is a machine learning algorithm that works a bit like that, but much more systematically.

Naive Bayes is a classification algorithm that makes predictions based on probabilities. The term “Naive” refers to the assumption that all features in a dataset are independent of each other. Though this assumption is often not true in real life, the Naive Bayes algorithm often works remarkably well. It is particularly popular in Natural Language Processing (NLP), especially for text classification. Despite its simplicity, it can be a powerful tool for solving complex problems.

Real-World Examples:

  • Spam Email Filtering: As mentioned, it’s widely used to identify spam emails based on words used in the email.
  • Sentiment Analysis: It can classify text as positive, negative, or neutral based on the words used.
  • Document Categorization: It can categorize documents into different categories (e.g., sports, politics, entertainment).
  • Medical Diagnosis: It can assist in medical diagnosis by using patient symptoms as features, and calculating the probability of them having a particular disease.

The Math Behind Naive Bayes

Naive Bayes relies on probability and Bayes’ theorem. Here’s a simplified explanation:

Bayes’ Theorem:

Bayes’ theorem calculates the probability of an event (a hypothesis) based on prior knowledge of conditions that might be related to the event.

Mathematically:

\[P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}\]

Where:

  • P(A|B) is the probability of event A happening, given that event B has happened. This is the posterior probability that the Naive Bayes calculates for every class.
  • P(B|A) is the probability of event B happening, given that event A has happened. This is called the likelihood.
  • P(A) is the probability of event A happening. This is called the prior probability.
  • P(B) is the probability of event B happening. This is called the evidence.
Bayes Theorem

Bayes Theorem Tree Diagram

Naive Bayes Assumption:

Naive Bayes assumes that all features are independent, which greatly simplifies the calculations.

Let’s say we want to determine if an email is spam or not. The email content would be the feature.

  • Let A be the event that the email is spam,
  • Let B be the event that the email contains specific word such as “lottery” Then P(A|B) is the probability that the email is spam, given that the email contains the word “lottery”.
  • P(B|A) is the likelihood which is the probability that the email contains the word “lottery” given the email is spam.
  • P(A) is the probability of email being spam in general. This value depends on the proportion of spam emails in the data and is known as prior probability.
  • P(B) is the probability of any email having the word “lottery”.

We calculate P(A|B) for each class and select the class which has maximum probability.

Example:

Suppose we’re classifying emails as “spam” or “not spam.” Let’s take the following simple data, and assume that we have following information about the data:

  1. Number of emails that are spam (A) = 2
  2. Number of emails that are not spam = 3
  3. Number of spam emails with the word “free” (B) = 1.
  4. Total number of emails = 5
  5. Number of normal emails with the word “free” = 1.
  6. Number of emails with “free” word = 2

We want to find P(spam| “free”) = P(B|A)*P(A)/P(B)

  • Prior Probability P(spam)= Number of spam emails/ Total Emails = 2/5
  • Likelihood P(“free” spam) = Number of spam emails with free / Total number of spam emails = 1/2
  • Evidence P(“free”) = Number of emails with free / Total number of emails = 2/5 P(spam|”free”) = (1/2)*(2/5)/(2/5) = 1/2 = 0.5 Therefore the probability of an email with the word “free” being spam is 0.5.

Multivariate Naive Bayes

If we have multiple features (for example the word “free”, and “lottery” in the email). Then the probability can be calculated using product as:

\[P(C | x_1, x_2, ..., x_n) = \frac{P(x_1 | C) * P(x_2 | C) *...*P(x_n | C) * P(C)}{P(x_1, x_2, ..., x_n)}\]

Where,

  • P(C|x1,x2, ... ,xn) is the probability of a given class C based on features x1, x2,...xn
  • P(xi|C) is the likelihood for feature xi, for a given class C.
  • P(C) is the prior probability.

Since denominator P(x1, x2, ..., xn) is a constant, it is ignored and for simplicity the formula is written as:

\[P(C | x_1, x_2, ..., x_n) \propto P(C) \prod_{i=1}^{n} P(x_i | C)\]

In practice, we would calculate this for every possible class and select the class with maximum value.

Prerequisites and Preprocessing

Assumptions:

  • Feature Independence: The core assumption is that all input features are independent of each other, which is often not the case in reality.
  • Categorical Features: Naive Bayes works well with categorical data, although the data can be converted to numerical data using different encoding schemes.
  • Numerical Features: For numerical features, Naive Bayes assumes that features follow a specific distribution, like Gaussian (normal), Multinomial, Bernoulli or others.

Preprocessing:

  • Categorical Data Encoding: Categorical features must be converted to numerical. Techniques like one-hot encoding, ordinal encoding, or feature hashing are common.
  • Text Data Vectorization: For text data, techniques like TF-IDF (Term Frequency-Inverse Document Frequency) or CountVectorizer are essential. These transform the text into numeric representations.
  • Handling Missing Values: Missing values are generally handled by imputation with a suitable value (e.g., mean or median) or removing samples with missing data. This step is not strictly required for the model to work, but can help improve the model performance.
  • Feature Scaling: Feature scaling is usually not required as Naive Bayes calculates probabilities with the actual feature values.

Python Libraries:

  • scikit-learn: Includes various Naive Bayes implementations such as GaussianNB, MultinomialNB, and BernoulliNB.
  • pandas: Used for data manipulation, loading data, and handling dataframes.
  • nltk: For text processing, you will need to pip install nltk, then download the necessary datasets using the code:

Tweakable Parameters and Hyperparameters

Naive Bayes algorithms have a small set of hyperparameters:

  • var_smoothing (GaussianNB): Adds a value to variance for smoothing, preventing zero variance.

    • Effect: Higher value reduces model complexity and generalization, prevents overfitting.

  • alpha (MultinomialNB, BernoulliNB): Smoothing parameter, also known as Laplace smoothing or Lidstone smoothing. This is an additive (Laplace/Lidstone) smoothing parameter that prevents the zero probability issue.

    • Effect: Higher value reduces model complexity and generalization, prevents overfitting. This helps prevent zero probabilities if a feature does not exist in a particular class in training data, but exist in the testing data.

  • binarize (BernoulliNB): Threshold for binarizing features.

    • Effect: If None, the input is presumed to already consist of binary vectors. If a number then it binarizes the feature on the given threshold.

  • fit_prior (all classes): Whether to learn class prior probabilities.

  • Effect: If False, a uniform prior will be used.

Data Preprocessing

Data preprocessing is very important for Naive Bayes, here are some key points:

  • Scaling is not required for the model as the model is based on probabilities and is not distance based.

  • Encoding of Categorical Data is important, and one hot encoding, ordinal encoding, feature hashing should be used for this purpose.

  • Text vectorization is required to convert the text into numerical features that Naive Bayes algorithm can understand.

  • Missing Value Imputation is required so that probabilities can be calculated correctly.

Examples:

  • Text data: If we want to classify emails into spam or not spam then that text data needs to be converted into numerical features.

  • Categorical data: If we want to predict if the user will purchase a product based on features such as “color,” then the color feature with values such as red, blue, green will need to be converted into numerical features using techniques such as one hot encoding, ordinal encoding etc.

  • Missing values: If any of the features is missing a particular value then that needs to be imputed with some reasonable value so that the model is able to perform calculations. For example we can impute the missing values with mean of the feature.

Implementation Example

Let’s implement a simple Multinomial Naive Bayes classifier for text data using scikit-learn with dummy data:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score, classification_report
import pickle
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer

# Download necessary nltk resources
nltk.download('punkt', quiet=True)
nltk.download('stopwords', quiet=True)
nltk.download('punkt_tab')

Output:

    [nltk_data] Downloading package punkt_tab to
    [nltk_data]    ...\nltk_data...
    [nltk_data]   Unzipping tokenizers\punkt_tab.zip.


# Sample data
data = {'text': ["This is a good movie",
                 "This is a bad movie",
                 "I like this food",
                 "I dislike that food",
                 "good job",
                 "bad luck",
                 "I love this",
                 "I hate that",
                 "this is a masterpiece",
                 "this is not good"],
        'label': ['positive', 'negative', 'positive', 'negative', 'positive', 'negative', 'positive', 'negative','positive','negative']}
df = pd.DataFrame(data)
df.head()

Output:

text label
0 This is a good movie positive
1 This is a bad movie negative
2 I like this food positive
3 I dislike that food negative
4 good job positive 
def preprocess_text(text):
    tokens = word_tokenize(text.lower())
    stop_words = set(stopwords.words('english'))
    stemmer = PorterStemmer()
    filtered_tokens = [stemmer.stem(token) for token in tokens if token.isalnum() and token not in stop_words]
    return " ".join(filtered_tokens)

df['text'] = df['text'].apply(preprocess_text)
df.head()

Output:

text label
0 good movi positive
1 bad movi negative
2 like food positive
3 dislik food negative
4 good job positive 
# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(df['text'], df['label'], test_size=0.3, random_state=42)


# Vectorize the text data
vectorizer = CountVectorizer()
X_train_vectorized = vectorizer.fit_transform(X_train)
X_test_vectorized = vectorizer.transform(X_test)


# Train a Multinomial Naive Bayes classifier
model = MultinomialNB()
model.fit(X_train_vectorized, y_train)

Output:

MultinomialNB()
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
# Make predictions
y_pred = model.predict(X_test_vectorized)

# Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
report = classification_report(y_test, y_pred)
print(f"Accuracy: {accuracy}")
print("Classification report: \n", report)

Output

    Accuracy: 0.3333333333333333
    Classification report: 
                   precision    recall  f1-score   support
    
        negative       0.00      0.00      0.00         2
        positive       0.33      1.00      0.50         1
    
        accuracy                           0.33         3
       macro avg       0.17      0.50      0.25         3
    weighted avg       0.11      0.33      0.17         3

# Save the model
filename = 'naive_bayes_model.pkl'
pickle.dump(model, open(filename, 'wb'))
pickle.dump(vectorizer, open("vectorizer.pkl", 'wb'))

# Load the model
loaded_model = pickle.load(open(filename, 'rb'))
loaded_vectorizer = pickle.load(open("vectorizer.pkl", 'rb'))
new_text = ["this was great", "this was horrible"]
new_text_vectorized = loaded_vectorizer.transform(new_text)
print("Loaded model prediction: ", loaded_model.predict(new_text_vectorized))

Output:

    Loaded model prediction:  ['positive' 'positive']

Explanation:

  • Accuracy: The accuracy score is approximately 0.33, or 33%. This indicates that the model correctly classified only one of the three instances in the test set. The model is not performing very well, this can also be due to random train test split. Accuracy is calculated as (TP+TN)/(TP+TN+FP+FN)

  • Classification Report:

    • Precision:

      • For the negative class, precision is 0.00, which means that the model was unable to correctly predict any negative class. TP/(TP+FP)

      • For the positive class, precision is 0.33 which is calculated as (1/3), meaning one third of predictions for positive class were actually correct. TP/(TP+FP)

    • Recall:

      • For the negative class, recall is 0.00. This means that the model was not able to correctly identify any of the actual negative instances. TP/(TP+FN)

      • For the positive class, recall is 1.00, which means that the model correctly identified all of the actual positive instances. TP/(TP+FN)

    • F1-score: The F1-score is 0.00 for the negative class and 0.50 for the positive class. The F1-score is the harmonic mean of precision and recall, and it is calculated as: 2*(Precision*Recall)/(Precision+Recall)

    • Support: The support shows the number of actual occurrences of that class in the test dataset. In the test set there are 2 instances of negative and 1 instance of positive class.

    • accuracy: This shows the accuracy of the model as a whole, and is equivalent to the value of the Accuracy output.

    • macro avg: Macro average of precision, recall and f1-score.

    • weighted avg: Weighted average of precision, recall, and f1-score, weighted by the support of each class.

  • Pickle: The model and vectorizer are saved using pickle.dump, and loaded using pickle.load.

  • Loaded model prediction: Output of the loaded model on new data points.

Post-Processing

  • Feature Importance: Although not as straightforward as in tree-based models, feature importance can be interpreted from the model’s parameters (probabilities of features for different classes).

  • AB Testing: If you have multiple ways to preprocess text or numeric features, AB testing can be done to find which preprocessing gives the best model.

  • Hypothesis Testing: Statistical significance of the model performance can be tested using hypothesis tests.

  • Other statistical tests can also be used for post processing if needed.

Hyperparameter Tuning

from sklearn.model_selection import GridSearchCV
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score, classification_report
import pickle
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer

# Download necessary nltk resources
nltk.download('punkt', quiet=True)
nltk.download('stopwords', quiet=True)

# Sample data
data = {'text': ["This is a good movie",
                 "This is a bad movie",
                 "I like this food",
                 "I dislike that food",
                 "good job",
                 "bad luck",
                 "I love this",
                 "I hate that",
                 "this is a masterpiece",
                 "this is not good"],
        'label': ['positive', 'negative', 'positive', 'negative', 'positive', 'negative', 'positive', 'negative','positive','negative']}
df = pd.DataFrame(data)

def preprocess_text(text):
    tokens = word_tokenize(text.lower())
    stop_words = set(stopwords.words('english'))
    stemmer = PorterStemmer()
    filtered_tokens = [stemmer.stem(token) for token in tokens if token.isalnum() and token not in stop_words]
    return " ".join(filtered_tokens)

df['text'] = df['text'].apply(preprocess_text)

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(df['text'], df['label'], test_size=0.3, random_state=42)


# Vectorize the text data
vectorizer = CountVectorizer()
X_train_vectorized = vectorizer.fit_transform(X_train)
X_test_vectorized = vectorizer.transform(X_test)

# Define hyperparameter grid
param_grid = {'alpha': [0.1, 0.5, 1.0, 2.0, 3.0]}

# Grid search for hyperparameter tuning
grid = GridSearchCV(MultinomialNB(), param_grid, cv=2)
grid.fit(X_train_vectorized, y_train)
print(f"Best parameters for Naive Bayes: {grid.best_params_}")
print("Best Score for Naive Bayes: ", grid.best_score_)

Output:

Best parameters for Naive Bayes: {'alpha': 2.0}
Best Score for Naive Bayes:  0.25
Accuracy: 0.3333333333333333
Classification report: 
               precision    recall  f1-score   support

    negative       0.00      0.00      0.00         2
    positive       0.33      1.00      0.50         1

    accuracy                           0.33         3
   macro avg       0.17      0.50      0.25         3
weighted avg       0.11      0.33      0.17         3

Checking Model Accuracy

Model accuracy can be checked using the following metrics:

  • Accuracy: Fraction of correct predictions. (TP+TN)/(TP+TN+FP+FN)

  • Precision: Measures the proportion of true positives among all positive predictions. TP/(TP+FP)

  • Recall (Sensitivity): Measures the proportion of true positives among all actual positives. TP/(TP+FN)

  • F1-Score: Harmonic mean of precision and recall. 2*(Precision*Recall)/(Precision+Recall).

  • AUC-ROC: (Area Under the Curve - Receiver Operating Characteristic). This metric is useful for checking the overall model quality.

  • Confusion Matrix: A table that summarizes the performance of a classification model.

Productionizing Steps

  • Local Testing: Create a test script to check that the model is loading correctly and giving the expected output.

  • On-Prem: Containerize the code and model using docker, and deploy on server.

  • Cloud: Deploy using cloud provider’s platform. Use cloud services to train, test and deploy.

  • Real time and Batch: Setup an ingestion pipeline for real time and batch prediction.

  • Monitoring: Monitor the model output for any deviations.

Conclusion

Naive Bayes is an efficient and effective algorithm, particularly when it comes to classifying text data. Its simplicity and speed make it a great choice when dealing with large text datasets.Despite the “naive” assumption of independence of features, the Naive Bayes algorithm is used effectively in various real world applications. However, more sophisticated models are replacing Naive Bayes, especially when dealing with numerical data. It is valuable for learning and understanding machine learning concepts.

References

  1. Scikit-learn Documentation: https://scikit-learn.org/stable/modules/naive_bayes.html

  2. NLTK Documentation: https://www.nltk.org/

  3. Wikipedia: https://en.wikipedia.org/wiki/Naive_Bayes_classifier