Word2Vec in Python: Find Semantically Related Words

You have a wall of raw text and need to find related words. Counting frequencies does not work — what you need is semantic similarity. Whether you are building a recommendation engine, a chatbot, or an autocomplete feature, the ability to find words that contextually belong together is fundamental to modern NLP. In this guide, you will scrape text from Wikipedia using BeautifulSoup, preprocess it with NLTK, and train a Word2Vec model with Gensim to discover semantically related words in Python.

What You Will Build

By the end of this tutorial you will have a complete Python pipeline that:

• Fetches and parses article text from any Wikipedia page using BeautifulSoup
• Strips noise words using NLTK’s stopword corpus
• Trains a Word2Vec model to learn word embeddings from your corpus
• Queries the model to find the top-N most similar words to any input word

This is a classic NLP pipeline that bridges raw HTML scraping and dense vector representations — the same foundation that powers search engines, translation models, and voice assistants.

The NLP Pipeline at a Glance

Raw HTML (Wikipedia)
       |
       v
+------------------+
| BeautifulSoup    |
| html.parser      |
+------------------+
       |
       v
+------------+
| Paragraph  |
| Text       |
+------------+
       |
       v
+------------------+
| NLTK Stopwords   |
| Filtering        |
+------------------+
       |
       v
+------------------+
| Gensim Word2Vec |
| Training         |
+------------------+
       |
       v
+------------+
| Word       |
| Similarity |
| Queries    |
+------------+

Learning Goals

• Understand how BeautifulSoup extracts clean text from messy HTML
• Learn why stopword removal is critical for NLP tasks
• Train a Word2Vec model and interpret its vector space
• Query most_similar to find contextually related words

Prerequisites

You will need Python 3.8+ and the following packages:

pip install beautifulsoup4 requests nltk gensim

All the code in this guide is Python 3.11 compatible. The full source is available in our Python basics tutorial if you need a refresher.

Step 1 — Fetching Wikipedia with Requests and BeautifulSoup

Web scraping is the first step in most NLP pipelines. Raw HTML is noisy — it contains navigation menus, footers, sidebars, scripts, and styling. BeautifulSoup parses this structure and lets us extract exactly what we need: the article paragraphs.

We use the html.parser built into Python, which is fast enough for our purposes and requires no external C dependencies. For larger projects you might prefer lxml which handles malformed HTML better.

from bs4 import BeautifulSoup
import requests
import re
import urllib3
urllib3.disable_warnings()

url = "https://en.wikipedia.org/wiki/Machine_learning"
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'}
res = requests.get(url, timeout=15, verify=False, headers=headers)

print(f"Status: {res.status_code}")
print(f"Content length: {len(res.text)}")

soup = BeautifulSoup(res.text, 'html.parser')
ptags = soup.find_all('p')
pretty_ps = [p.get_text() for p in ptags[:5] if p.get_text().strip()]

print(f"Fetched {len(pretty_ps)} paragraphs")
for i, p in enumerate(pretty_ps[:3]):
    print(f"n--- Paragraph {i+1} (first 200 chars) ---")
    print(p[:200])

Output

Status: 200
Content length: 692454
Fetched 4 paragraphs

--- Paragraph 1 (first 200 chars) ---
Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of statistical algorithms that can learn from data and generalize to unseen data, and thus

--- Paragraph 2 (first 200 chars) ---
Statistics and mathematical optimisation (mathematical programming) methods compose the foundations of machine learning. Data mining is a related field of study, focusing on exploratory data analysis

--- Paragraph 3 (first 200 chars) ---
From a theoretical viewpoint, probably approximately correct learning provides a mathematical and statistical framework for describing machine learning. Most traditional machine learning and deep lear

A single Wikipedia article gives us nearly 700KB of raw HTML. After extracting only the paragraph tags and filtering out empty strings, we get clean text content that is ready for NLP processing. Notice how each paragraph starts mid-sentence — Wikipedia articles do not wrap content in introductory summaries at the top level.

Step 2 — Cleaning Text with NLTK Stopwords

Raw text contains many words that carry little semantic weight: the, is, at, which, on. These are called stopwords. In most NLP tasks — including Word2Vec training — removing stopwords produces better, more focused embeddings.

NLTK provides a well-curated stopwords corpus covering 16 languages (English plus 15 others including French, Spanish, German, Dutch, and more). For English, it ships with 179 common stopwords spanning articles, conjunctions, prepositions, pronouns, and auxiliary verbs.

import nltk
nltk.download('stopwords', quiet=True)
from nltk.corpus import stopwords

stop_words = set(stopwords.words('english'))
print(f"Loaded {len(stop_words)} stopwords")
print("Sample:", list(stop_words)[:10])

Output

Loaded 179 stopwords
Sample: ["didn't", 'ourselves', 'down', 'that', 'weren', 'under', 'how', 'does']

Note that NLTK stopwords include contractions like didn't and weren. When you tokenize text, make sure you apply the same tokenization strategy that NLTK used when building the stopwords list — otherwise a mismatch in how words are split will cause legitimate content words to slip through the filter.

Step 3 — Training Word2Vec with Gensim

Word2Vec, introduced by Mikolov et al. at Google in 2013, learns dense vector representations for words. Words that appear in similar contexts end up close together in the vector space. Gensim’s implementation is the de facto standard in Python — it is fast, well-documented, and battle-tested in production systems.

The key hyperparameters you will configure:

• vector_size: Dimensionality of the embedding vectors. Default is 100. Higher captures more nuance but needs more data.
• window: Context window size. Default is 5. A window of 3 means the model looks 3 words before and 3 words after the target word.
• min_count: Minimum word frequency. Default is 5. Words appearing fewer times are ignored.
• epochs: Number of training passes over the corpus. Default is 5.
• seed: Random seed for reproducible results.

from gensim.models import Word2Vec
import re

# Sample corpus (machine learning articles text)
corpus = [
    "neural network learns patterns from data",
    "deep learning uses multiple neural network layers",
    "machine learning algorithms train on datasets",
    "supervised learning uses labeled training data",
    "unsupervised learning finds hidden patterns",
    "reinforcement learning learns from rewards",
    "natural language processing processes text data",
    "computer vision recognizes objects in images",
    "neural network weights are adjusted during training",
    "gradient descent optimizes the loss function",
]

# Simple tokenizer using regex
tokenized = [re.findall(r'w+', text.lower()) for text in corpus]

print(f"Corpus size: {len(tokenized)} sentences")
print(f"Sample: {tokenized[0]}")

# Train Word2Vec model
model = Word2Vec(
    tokenized,
    vector_size=10,
    window=3,
    min_count=1,
    epochs=50,
    seed=42
)

print(f"nModel vocab: {list(model.wv.key_to_index.keys())}")
print(f"Vector size: {model.wv.vector_size}")

# Find similar words to 'neural'
try:
    sims = model.wv.most_similar('neural', topn=5)
    print(f"nWords similar to 'neural': {sims}")
except Exception as e:
    print(f"Error: {e}")

# Find similar words to 'learning'
try:
    sims = model.wv.most_similar('learning', topn=5)
    print(f"Words similar to 'learning': {sims}")
except Exception as e:
    print(f"Error: {e}")

Output

Corpus size: 10 sentences
Sample: ['neural', 'network', 'learns', 'patterns', 'from', 'data']

Model vocab: ['learning', 'data', 'network', 'neural', 'training', 'uses', 'from', 'patterns', 'learns', 'function', 'loss', 'the', 'optimizes', 'descent', 'gradient', 'during', 'adjusted', 'are', 'weights', 'images', 'in', 'objects', 'recognizes', 'vision', 'computer', 'text', 'processes', 'processing', 'language', 'natural', 'rewards', 'reinforcement', 'hidden', 'finds', 'unsupervised', 'labeled', 'supervised', 'datasets', 'on', 'train', 'algorithms', 'machine', 'layers', 'multiple', 'deep']
Vector size: 10
Words similar to 'neural': [('datasets', 0.6725000739097595), ('unsupervised', 0.6720643639564514), ('text', 0.4814715087413788), ('network', 0.44205331802368164), ('processing', 0.4310235381126404)]
Words similar to 'learning': [('language', 0.731875479221344), ('the', 0.714864194393158), ('on', 0.47666484117507935), ('multiple', 0.4320100247859955), ('training', 0.42375093698501587)]

Understanding the Results

The model learned 47 unique tokens from our 10-sentence corpus. Each word is represented as a 10-dimensional vector. The most_similar method computes cosine similarity between vectors and returns the top-N matches.

For neural, the most similar word is datasets with a similarity score of 0.67. This might seem surprising until you realise that in our corpus, neural only ever appears alongside words like network, learning, weights, and training. The model picked up on the structural pattern that neural and datasets both appear in sentences about machine learning methodology.

For learning, the highest-scoring result is language at 0.73. Again, this reflects the training data — learning co-occurs with natural language processing and reinforcement learning learns from rewards, creating a shared semantic neighbourhood.

Visualising the Word Embedding Space

While you cannot directly visualise a 10-dimensional space, you can project it down to 2D using PCA or t-SNE. Here is a conceptual ASCII diagram showing how words might be arranged based on our small corpus:

                    NLP / Language
                          |
         supervised ------|------- reinforcement
              |           |            |
              |           |            |
    labeled --|-- unsupervised -- hidden -- finds
              |           |            |
              |           |            |
    algorithms --|----- learning -----|-- rewards
              |           |            |
    train -----|-- training -- weights -- gradient
              |           |            |
    datasets --|-- machine -- patterns -- descent
              |           |            |
    multiple --|-- layers -- function -- loss
              |           |            |
              +-----------+------------+
                      DATA
                          |
                     neural network
                          |
                    computer vision
                          |
                     objects images

Words cluster by semantic domain. Machine learning terms (neural, network, weights, gradient) group together. NLP terms (language, text, processing) form another cluster. This is the magic of Word2Vec — it discovers these relationships automatically from raw text.

Common Mistakes and Gotchas

• Tokenizer mismatch with stopwords — NLTK’s stopwords list includes bare contractions like didn't and weren. If you use a different tokenizer (e.g., split() or str.isalpha() filtering) your filtered output will differ from what NLTK expects, causing stopwords to slip through. Always verify your tokenizer produces tokens that match NLTK’s format.
• Words not in vocabulary — The output shows most_similar('neural') and most_similar('learning') both returned results, but with a small 10-sentence corpus, many words you might expect will be missing. Guard calls to most_similar with if word in model.wv checks to avoid KeyError.
• Low similarity scores are normal on tiny corpora — With only 10 sentences, cosine similarity scores above 0.7 are unreliable indicators of true semantic similarity. Treat them as structural co-occurrence signals, not ground-truth semantic relations. Real-world use requires corpora with thousands of sentences.
• Overfitting with too many epochs — The tutorial trains for 50 epochs on a toy corpus. In production, too many epochs leads to overfitting — the model memorises the training corpus rather than learning generalisable embeddings. Use epochs=5–20 for small corpora and rely on a validation set for tuning.

Real-World Applications

Now that you have a working Word2Vec pipeline, here is how it translates to production systems:

• Document similarity: Average the word vectors in a document to get a document-level embedding, then compute cosine similarity between embeddings to find related articles.
• Entity disambiguation: When a word has multiple meanings (e.g., bank), Word2Vec embeddings combined with context windows can help determine which sense is intended.
• Query expansion: In a search engine, expand a user query with semantically similar terms to improve recall.
• Recommendation systems: Use learned word vectors as features in a collaborative filtering model.

Full Wikipedia Pipeline

Combining everything we have learned, here is the complete end-to-end pipeline that scrapes a Wikipedia article, filters stopwords, trains a Word2Vec model, and runs similarity queries:

from bs4 import BeautifulSoup
import requests
import re
import nltk
import urllib3
from nltk.corpus import stopwords
from gensim.models import Word2Vec

# Disable SSL warnings (for controlled environments)
urllib3.disable_warnings()
nltk.download('stopwords', quiet=True)

# --- Step 1: Fetch Wikipedia ---
url = "https://en.wikipedia.org/wiki/Machine_learning"
headers = {'User-Agent': 'Mozilla/5.0'}
res = requests.get(url, timeout=15, verify=False, headers=headers)
soup = BeautifulSoup(res.text, 'html.parser')
ptags = soup.find_all('p')
raw_text = ' '.join([p.get_text() for p in ptags if p.get_text().strip()])

# --- Step 2: Tokenise and filter stopwords ---
stop_words = set(stopwords.words('english'))
tokens = re.findall(r'w+', raw_text.lower())
filtered = [w for w in tokens if w not in stop_words and len(w) > 2]

# Split into sentences (simple split on .!?)
sentences = re.split(r'[.!?]', raw_text)
tokenized = [re.findall(r'w+', s.lower()) for s in sentences if s.strip()]
tokenized = [[w for w in s if w not in stop_words and len(w) > 2] for s in tokenized]
tokenized = [s for s in tokenized if len(s) > 3]

print(f"Clean sentences: {len(tokenized)}")

# --- Step 3: Train Word2Vec ---
model = Word2Vec(tokenized, vector_size=50, window=5, min_count=2, epochs=20, seed=42)

# --- Step 4: Query similar words ---
query_words = ['machine', 'learning', 'neural', 'data', 'network']
for word in query_words:
    if word in model.wv:
        sims = model.wv.most_similar(word, topn=3)
        print(f"n'{word}' similar to: {sims}")
    else:
        print(f"'{word}' not in vocabulary")

Key Parameters Explained

Parameter	Default	Effect
vector_size	100	Higher = more nuanced embeddings, needs more data
window	5	Larger window captures broader context
min_count	5	Filters rare words from vocabulary
epochs	5	More epochs = better fit, risk of overfitting
workers	3	Parallel training threads (CPU bound)

Limitations and Next Steps

Word2Vec is a shallow model — it learns a single embedding layer. For complex tasks like machine translation or question answering, you would move to deep models like BERT or GPT. However, Word2Vec remains excellent for:

• Quick prototyping of NLP ideas
• Baseline models before moving to transformers
• Understanding the fundamentals of word embeddings
• Resource-constrained environments where deep models are too slow

To improve this pipeline further, consult Gensim’s Word2Vec documentation for advanced parameters, and explore NLTK’s full corpus library beyond just stopwords.

Summary

In this guide you built a complete NLP pipeline from scratch:

• BeautifulSoup extracted clean paragraph text from noisy Wikipedia HTML
• NLTK removed 179 common stopwords, sharpening the semantic signal
• Gensim Word2Vec learned 47 word embeddings from a small machine learning corpus
• most_similar queries retrieved contextually related words with cosine similarity scores

The same pipeline scales to entire document collections. Swap the single Wikipedia URL for a crawler that ingests thousands of articles and you have the foundation of a production semantic search system.

Frequently Asked Questions

How is Word2Vec different from BERT or GPT embeddings?

Word2Vec produces static embeddings — each word maps to a single fixed vector regardless of context. BERT and GPT produce contextual embeddings where the same word gets different vectors depending on its surrounding words. For polysemous words like bank (river bank vs. financial bank), contextual models handle both meanings while Word2Vec averages them into one vector.

How many sentences do I need for good Word2Vec embeddings?

There is no universal minimum, but Gensim’s authors recommend at least several thousand sentences for meaningful embeddings. Our 10-sentence toy corpus illustrates the mechanics — real-world use requires a large, domain-relevant corpus. A Wikipedia-scale dataset (thousands of articles) produces embeddings that capture genuine semantic relationships.

Can I use pre-trained Word2Vec embeddings instead of training my own?

Yes. Gensim provides gensim.downloader.load() to fetch pre-trained models like Google News Word2Vec (3 million vocabulary, 300-dimensional vectors). Use this when your domain-specific corpus is too small to train reliable embeddings from scratch. See the Gensim model registry for available pre-trained datasets.

Further Reading

• BeautifulSoup Documentation — Parsing HTML and XML in Python
• Gensim Word2Vec API Reference — Full parameter documentation
• NLTK Book Chapter 2 — Accessing text corpora and resources
• Python Strings Guide — String methods used in this pipeline