📊 Master Sequence Labeling: A Guide to Conditional Random Fields (CRF)

February 17, 2021 by dumira

In traditional machine learning, we usually treat classification tasks as isolated events. If you train a model to predict whether an email is spam, it looks at that email in a vacuum.

However, when dealing with sequential data—like text sentences, DNA strands, or time-series logs—order matters. A word’s meaning depends heavily on its neighbors. To solve this, we use sequence labeling algorithms, and Conditional Random Fields (CRF) remain one of the most powerful, interpretability-friendly statistical frameworks for the job.

In this guide, we’ll dive into how the CRF algorithm works conceptually and build a step-by-step implementation in Python for a Named Entity Recognition (NER) task.

🏗️ What is a Conditional Random Field?

A Conditional Random Field (CRF) is a discriminative undirected graphical model. Unlike independent classifiers, a CRF predicts a label sequence $y = (y_1, y_2, \dots, y_m)$ for an entire input sequence $x = (x_1, x_2, \dots, x_m)$ globally rather than making isolated token-by-token decisions.

HMM vs. MEMM vs. CRF

To understand why CRFs excel, it helps to look at their evolutionary predecessors:

  • Hidden Markov Models (HMM): Generative models that calculate the joint probability $P(x, y)$. Because they model the generation of the text, they struggle to natively handle overlapping or contextual features (e.g., checking if the next word starts with a capital letter).
  • Maximum Entropy Markov Models (MEMM): Discriminative models that compute $P(y|x)$ but make local transitions at each step. This leads to the infamous label bias problem, where states with low-entropy out-degree transitions dominate the path selection regardless of the global sequence context.
  • Conditional Random Fields (CRF): Solves the label bias problem by normalizing the probabilities globally over the entire sequence path. It calculates the conditional probability $P(y|x)$ globally, striking a perfect balance between contextual feature flexibility and global sequence awareness.

🛠️ Step-by-Step Python Implementation

We will implement a Linear-Chain CRF using the popular sklearn-crfsuite package to extract named entities (like locations or organizations) from tokenized sentences.

Step 1: Install Dependencies

Open your terminal and install the required sequence labeling library:

Bash

pip install sklearn-crfsuite

Step 2: Prepare Mock Training Data

In sequence labeling, data is typically structured as sequences of tuples containing (Word, POS_Tag, Entity_Tag). Let’s set up a small annotated corpus using the popular IOB (Inside, Outside, Beginning) format:

Python

training_corpus = [
    [('London', 'NNP', 'B-LOC'), ('is', 'VBZ', 'O'), ('the', 'DT', 'O'), ('capital', 'NN', 'O'), ('of', 'IN', 'O'), ('England', 'NNP', 'B-LOC'), ('.', '.', 'O')],
    [('Alice', 'NNP', 'B-PER'), ('works', 'VBZ', 'O'), ('at', 'IN', 'O'), ('Google', 'NNP', 'B-ORG'), ('in', 'IN', 'O'), ('New', 'NNP', 'B-LOC'), ('York', 'NNP', 'I-LOC'), ('.', '.', 'O')]
]

Step 3: Engineer Feature Extractors

The superpower of a CRF is its reliance on custom-defined feature functions. We write a function that extracts rich structural details about the target word, its prefix/suffix, its casing, and its immediate neighbors:

Python

def word2features(sentence, i):
    word = sentence[i][0]
    postag = sentence[i][1]

    # Features for the current word
    features = {
        'bias': 1.0,
        'word.lower()': word.lower(),
        'word[-3:]': word[-3:],
        'word[-2:]': word[-2:],
        'word.isupper()': word.isupper(),
        'word.istitle()': word.istitle(),
        'word.isdigit()': word.isdigit(),
        'postag': postag,
    }
    
    # Features for the previous word (Contextual)
    if i > 0:
        word1 = sentence[i-1][0]
        postag1 = sentence[i-1][1]
        features.update({
            '-1:word.lower()': word1.lower(),
            '-1:word.istitle()': word1.istitle(),
            '-1:postag': postag1,
        })
    else:
        features['BOS'] = True # Beginning of Sentence

    # Features for the next word (Contextual)
    if i < len(sentence) - 1:
        word1 = sentence[i+1][0]
        postag1 = sentence[i+1][1]
        features.update({
            '+1:word.lower()': word1.lower(),
            '+1:word.istitle()': word1.istitle(),
            '+1:postag': postag1,
        })
    else:
        features['EOS'] = True # End of Sentence

    return features

# Helper maps to format full sequences
def sent2features(sent):
    return [word2features(sent, i) for i in range(len(sent))]

def sent2labels(sent):
    return [label for token, postag, label in sent]

Step 4: Extract Features & Format Data

We map our raw sequence datasets through our feature functions to generate the distinct dictionaries ($X$) and parallel ground-truth target sequence lists ($y$) that the model expects.

Python

X_train = [sent2features(s) for s in training_corpus]
y_train = [sent2labels(s) for s in training_corpus]

Step 5: Initialize and Train the CRF Tagger

We instantiate the CRF class using the standard L-BFGS optimization algorithm. We also supply $L_1$ (c1) and $L_2$ (c2) regularization multipliers to keep our feature weights sparse and prevent overfitting.

Python

import sklearn_crfsuite

# Initialize the model
crf = sklearn_crfsuite.CRF(
    algorithm='lbfgs',
    c1=0.1,
    c2=0.1,
    max_iterations=100,
    all_possible_transitions=True
)

# Fit the sequence model
print("🏋️‍♂️ Training the CRF model...")
crf.fit(X_train, y_train)
print("✅ Training complete!")

Step 6: Inference on New Sequences

Let’s construct a brand new sequence that the model has never seen before, generate its matching features, and run our global sequence prediction:

Python

# Unseen text: "Bob works in London ."
test_sentence = [('Bob', 'NNP'), ('works', 'VBZ'), ('in', 'IN'), ('London', 'NNP'), ('.', '.')]

# Generate features (simulating missing entity tags)
X_test = [word2features([(w, p, 'O') for w, p in test_sentence], i) for i in range(len(test_sentence))]

# Predict paths
predictions = crf.predict_single(X_test)

for word, tag in zip([w for w, p in test_sentence], predictions):
    print(f"{word} -> {tag}")

Expected Output:

Bob -> B-PER

works -> O

in -> O

London -> B-LOC

. -> O

📈 When Should You Choose CRFs Over Deep Learning?

With the dominance of Transformers and LLMs, why look at CRFs?

  • Extreme Efficiency: CRFs train in seconds or minutes on a standard CPU, unlike heavy neural models that demand persistent GPU clusters.
  • Deterministic Transparency: You can explicitly pull the transition matrix to view the exact log-likelihood mathematical weights associated with stepping from B-LOC to I-LOC.
  • Hybrid Layouts (BiLSTM-CRF): In advanced settings, researchers place a CRF layer on top of a BiLSTM or BERT representation layer. The deep learning layers extract high-level contextual vectors, while the final CRF layer forces strict global sequence sanity rules (e.g., ensuring an I-PER label never follows an O label directly).

CRFs remain a premier choice for lightweight production deployments, structural feature engineering, and high-speed text parsing.