Automate Keyword Clustering by Search Intent with Python

Understanding search intent is crucial for effective SEO. This guide demonstrates how to automate keyword clustering by search intent using Python, allowing you to scale your keyword research and gain valuable insights.

Why Automate Search Intent Clustering?

With the rise of AI-powered search, efficient keyword grouping is more critical than ever. This method leverages Google's own AI through SERP analysis, eliminating the need to build your own AI models.

Leveraging SERPs for Search Intent Insights

This approach assumes Google ranks URLs based on user query satisfaction. Therefore, similar SERPs suggest similar search intent. Automating this comparison provides speed and precision.

Python-Based Keyword Clustering (With Code)

1. Import SERP Data

Begin by importing your SERP data from a CSV file into a Pandas DataFrame.

import pandas as pd
import numpy as np
serps_input = pd.read_csv('data/sej_serps_input.csv')
del serps_input['Unnamed: 0']

2. Filter for Page 1 Results

Focus on Page 1 results for accurate intent analysis. Filter the DataFrame at the keyword level.

# Code to filter for Page 1 results (provided in original content)

3. Convert URLs to a String

Compress the URLs for each keyword into a single string representing the keyword's SERP.

# Code to convert URLs to a string (provided in original content)

4. Compare SERP Similarity

Compare every keyword SERP pair using a custom function that considers both URL overlap and order.

# Code to compare SERP similarity (provided in original content)

5. Cluster Keywords

Group keywords with a similarity score above a defined threshold (e.g., 40%).

# Code to cluster keywords (provided in original content)

Utilizing Clustered Keywords for SEO

These clustered keywords can enhance various aspects of your SEO strategy:

  • Improve SEO reporting and trend analysis.
  • Structure paid search campaigns for higher Quality Scores.
  • Optimize e-commerce site taxonomy and facet navigation.

Download the full code: Python Keyword Clustering Code

Further Resources: