SEO Developer Course & Resource Guide

Technology: seo · Category: tooling · Last reviewed: 2026-08-23

Source: https://tech-stack.codeamanilabs.org/guide/seo

Insight:

SEO is now two audiences. Search engines crawl → index → rank, gated by robots.txt, sitemap.xml, on-page signals (title/meta/canonical), structured data (JSON-LD), and Core Web Vitals. LLMs (ChatGPT, Claude, Perplexity) increasingly answer for users — and llms.txt is the emerging standard for feeding them a curated, low-token digest of your site. Win both: ship the root files, mark up content with schema.org, pass the Lighthouse SEO audit, and keep LCP ≤ 2.5s. robots.txt is not an index blocker — that's noindex. All file templates below are copy-paste ready.

███████╗███████╗ ██████╗
██╔════╝██╔════╝██╔═══██╗
███████╗█████╗  ██║   ██║
╚════██║██╔══╝  ██║   ██║
███████║███████╗╚██████╔╝
╚══════╝╚══════╝ ╚═════╝

SEO Developer Course & Resource Guide

Focus: A developer's hands-on path to being found — by search engines and by LLMs. Crawlability files (robots.txt, sitemap.xml, llms.txt), on-page signals, structured data, Core Web Vitals, the Lighthouse SEO audit, AI discoverability (GEO), measurement, and a Next.js reference implementation. Grounded in developers.google.com, developer.chrome.com, sitemaps.org, llmstxt.org, and schema.org; reviewed 2026-08-23.

How this course works

Six parts, each ending with a ✅ capability checkpoint and a 🛠 exercise. The interactive learn module above this page — a live SERP + social card + structured-data previewer — is your illustration for Parts 2–3; edit a title/description there and watch the search snippet and length warnings update. Every file you need is in §Templates.

Table of Contents

1. How discovery works — crawl → index → rank → cite
2. Crawlability files — robots.txt · sitemap.xml · llms.txt
3. On-page SEO — titles, meta, canonical, Open Graph
4. Structured data — JSON-LD + rich results
5. Technical SEO & performance — Core Web Vitals, Lighthouse SEO audit
6. AI discoverability (GEO) — ranking in LLMs
7. Measurement · Next.js implementation · Templates · codeAmani notes

Official Documentation

1. How discovery works: crawl → index → rank → cite

flowchart LR
  C["Crawl<br/>robots.txt"] --> I["Index<br/>sitemap + clean HTML"]
  I --> R["Rank<br/>signals + Core Web Vitals"]
  R --> S["SERP result"]
  C --> L["llms.txt"]
  L --> RT["LLM retrieves"]
  RT --> CT["Cited in AI answer"]

SEARCH ENGINES                                   LLMs (new)
 crawl ──▶ index ──▶ rank ──▶ SERP               retrieve ──▶ synthesize ──▶ cite
   │         │         │                              │
 robots.txt  sitemap   on-page signals +          llms.txt + clean content +
 (may/can't) (what     structured data +          structured data + being
             exists)   Core Web Vitals            quotable & authoritative

• Crawl — bots fetch pages. robots.txt says where they may go (not what gets hidden).
• Index — the engine stores and understands the page (helped by structured data + clean HTML).
• Rank — relevance + quality + page experience (Core Web Vitals) decide ordering.
• Cite (new) — LLMs retrieve and quote sources. llms.txt, markdown-clean content, and clear authority make you the cited answer.

💡 The mindset shift: you're no longer optimising only for ten blue links. You're optimising to be the answer — in a SERP rich result and in a chatbot's cited response.

2. Crawlability: the root files

Three files at your domain root, each for a different reader.

robots.txt — where crawlers may go

Lives at /robots.txt. Controls crawl traffic; it is not a way to hide a page from the index — a blocked-but-linked page can still appear (use noindex to truly exclude).

# /robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

# Point crawlers (and many AI bots) at your sitemap
Sitemap: https://example.com/sitemap.xml

sitemap.xml — what exists

Lives at /sitemap.xml. Lists canonical URLs so crawlers discover everything. Namespace http://www.sitemaps.org/schemas/sitemap/0.9; <loc> is required, the rest optional. Limits: 50,000 URLs / 50 MB per file — beyond that, split and use a sitemap index.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-06-07</lastmod>
    <changefreq>weekly</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/pricing</loc>
    <lastmod>2026-06-01</lastmod>
  </url>
</urlset>

llms.txt — a digest for LLMs

The emerging standard (llmstxt.org) for feeding LLMs a curated, low-token map of your site at /llms.txt. Markdown, in this exact order: an H1 name (only required part), a blockquote summary, optional body, then H2 sections of [link](url): description lists, with a skippable ## Optional section.

# codeAmani Labs

> AI-augmented products for the Kenyan and East African market — M-Pesa payments,
> Next.js apps, and developer tooling.

codeAmani builds mobile-first SaaS with M-Pesa as the default payment rail.

## Docs
- [Tech stack](https://example.com/stack.md): Our stack and conventions
- [M-Pesa integration](https://example.com/mpesa.md): Daraja STK Push lifecycle

## Optional
- [Blog](https://example.com/blog.md): Background articles

Also serve markdown versions of pages (append .md to a URL) and optionally an llms-full.txt (everything concatenated) so an LLM can ingest clean content without parsing HTML.

✅ Checkpoint: You can write a correct robots.txt, a valid sitemap, and an llms.txt — and you know robots.txt ≠ noindex. 🛠 Exercise: Write all three for a 3-page site, then validate the sitemap in Google Search Console.

3. On-page SEO: the signals on every page

<head>
  <!-- Title: the SERP headline + browser tab. Front-load the keyword; ~50–60 chars. -->
  <title>M-Pesa Integration Guide | codeAmani Labs</title>

  <!-- Meta description: the SERP snippet. ~150–160 chars, compelling, unique per page. -->
  <meta name="description" content="Integrate M-Pesa STK Push with Daraja in Next.js — auth, callbacks, and idempotency, with copy-paste code.">

  <!-- Canonical: the one true URL for this content (kills duplicate-content dilution). -->
  <link rel="canonical" href="https://example.com/mpesa">

  <!-- Crawl directives (per-page; THIS is how you exclude from the index). -->
  <meta name="robots" content="index, follow">

  <!-- Open Graph — the social/link-preview card (LinkedIn, WhatsApp, Slack). -->
  <meta property="og:title" content="M-Pesa Integration Guide">
  <meta property="og:description" content="Daraja STK Push in Next.js, step by step.">
  <meta property="og:image" content="https://example.com/og/mpesa.png">
  <meta property="og:type" content="article">
  <meta property="og:url" content="https://example.com/mpesa">

  <!-- Twitter/X card -->
  <meta name="twitter:card" content="summary_large_image">

  <meta name="viewport" content="width=device-width, initial-scale=1">
</head>

Also on every page: one <h1>, a logical heading outline (h2/h3), descriptive alt on images, descriptive link text (not "click here"), semantic HTML (<nav>, <main>, <article>), and hreflang if you serve multiple languages/regions.

✅ Checkpoint: Every page has a unique title + description, a canonical, OG tags, one h1, and alt text.

4. Structured data (JSON-LD)

Machine-readable facts about the page that earn rich results (stars, FAQs, breadcrumbs) in search and feed LLMs clean entities. Google recommends JSON-LD in a <script>. Validate with the Rich Results Test.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "M-Pesa Integration Guide",
  "author": { "@type": "Organization", "name": "codeAmani Labs" },
  "datePublished": "2026-06-07",
  "image": "https://example.com/og/mpesa.png"
}
</script>

Common high-value types: Article, BreadcrumbList, FAQPage, Product (with Offer + AggregateRating), Organization, LocalBusiness, HowTo, WebSite (with SearchAction). Use schema.org for the full vocabulary; type your objects with schema-dts in TypeScript.

// FAQPage — earns an expandable FAQ rich result
{
  "@context": "https://schema.org", "@type": "FAQPage",
  "mainEntity": [{ "@type": "Question", "name": "Does it support M-Pesa?",
    "acceptedAnswer": { "@type": "Answer", "text": "Yes — Daraja STK Push is first-class." } }]
}

✅ Checkpoint: Your key pages carry valid JSON-LD that passes the Rich Results Test.

5. Technical SEO & performance

Core Web Vitals (a ranking signal)

Page experience counts. Targets at the 75th percentile (see the chrome-devtools guide):

The Lighthouse SEO audit

Run it (DevTools → Lighthouse, or lhci). The SEO category scans automatable signals — each weighted equally (except the manual structured-data check):

• Document has a <title> and a <meta name="description">
• Page is crawlable (not blocked by robots/noindex when it should be indexed)
• Has a valid rel=canonical
• Links have descriptive text; image elements have alt
• Valid hreflang (if used); has a viewport; returns a successful HTTP status
• robots.txt is valid

# CI gate — fail the PR if SEO regresses (see github guide for the workflow)
npx lighthouse https://example.com --only-categories=seo,performance --output json

✅ Checkpoint: Lighthouse SEO ≥ 0.9 and LCP ≤ 2.5 s on mobile.

6. AI discoverability (GEO — ranking in LLMs)

"Generative Engine Optimization": being the source an LLM retrieves and cites. Search and LLM optimisation overlap, but LLMs reward different things:

• Serve llms.txt (and .md page versions) so models ingest curated, low-token content without HTML noise.
• Be quotable — clear claims, definitions, and self-contained sections an LLM can lift verbatim.
• Structured data doubles as LLM food — JSON-LD entities are clean, unambiguous facts.
• Authority + freshness — accurate lastmod, named authors (Organization/Person schema), and content that's actually correct (LLMs are increasingly grounded and fact-checked).
• Don't accidentally block AI bots — decide deliberately in robots.txt whether to allow GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.

# robots.txt — example: allow search + select AI crawlers, block one
User-agent: GPTBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

✅ Checkpoint: You serve llms.txt + markdown content and have made an explicit allow/deny decision for AI crawlers.

7. Measurement

import { onLCP, onINP, onCLS } from "web-vitals";
const send = (m: unknown) => navigator.sendBeacon("/vitals", JSON.stringify(m));
onLCP(send); onINP(send); onCLS(send);   // report real-user vitals

Metrics that matter (in order): indexed pages → impressions → average position → CTR (driven by title/description) → conversions. Vanity keyword rankings are downstream of these.

8. Next.js reference implementation

Next.js (App Router, the codeAmani default — verified on v15/v16; the Metadata API is identical across both, current stable 16.3.2) generates the files and tags natively.

// app/robots.ts → serves /robots.txt
import type { MetadataRoute } from "next";
export default function robots(): MetadataRoute.Robots {
  return {
    rules: [{ userAgent: "*", allow: "/", disallow: ["/admin/", "/api/"] }],
    sitemap: "https://example.com/sitemap.xml",
  };
}

// app/sitemap.ts → serves /sitemap.xml
import type { MetadataRoute } from "next";
export default function sitemap(): MetadataRoute.Sitemap {
  return [
    { url: "https://example.com/", lastModified: new Date(), changeFrequency: "weekly", priority: 1 },
    { url: "https://example.com/pricing", lastModified: new Date() },
  ];
}

// app/mpesa/page.tsx → per-page metadata (title, description, canonical, OG)
import type { Metadata } from "next";
export const metadata: Metadata = {
  title: "M-Pesa Integration Guide | codeAmani Labs",
  description: "Integrate M-Pesa STK Push with Daraja in Next.js — auth, callbacks, idempotency.",
  alternates: { canonical: "https://example.com/mpesa" },
  openGraph: { title: "M-Pesa Integration Guide", images: ["/og/mpesa.png"], type: "article" },
  twitter: { card: "summary_large_image" },
};

// JSON-LD in a Server Component — render the script inline
export default function Page() {
  const ld = { "@context": "https://schema.org", "@type": "Article", headline: "M-Pesa Integration Guide" };
  return <script type="application/ld+json" dangerouslySetInnerHTML={{ __html: JSON.stringify(ld) }} />;
}

For llms.txt, add a route (app/llms.txt/route.ts) or a static file in public/. next-sitemap automates large/multi-sitemap setups.

Copy-paste file templates

Everything you need, ready to drop into any codeAmani project (replace example.com):

• robots.txt → §2 (+ the AI-crawler variant in §6)
• sitemap.xml → §2
• llms.txt → §2
• Meta + Open Graph <head> block → §3
• JSON-LD (Article / FAQPage) → §4
• Next.js robots.ts / sitemap.ts / metadata → §8

Troubleshooting

codeAmani notes

• Two audiences, both first-class. Ship robots.txt, sitemap.xml, and llms.txt on every codeAmani site — we want to rank in Google and be the cited answer in ChatGPT/Claude/Perplexity. Decide AI-crawler access deliberately.
• Performance is SEO here. Our users are on 2G/3G; LCP ≤ 2.5s on mobile is both a ranking signal and a UX necessity. Gate it in CI with Lighthouse (see chrome-devtools + github guides).
• Local + multilingual. Use LocalBusiness schema (Nairobi address, hours), hreflang for English/Swahili variants, and Google Business Profile for local intent ("M-Pesa developer Nairobi").
• Product schema for M-Pesa commerce. Mark up priced products with Product + Offer (currency KES) so rich results show price/availability.
• Secrets stay server-side. SEO is public by nature, but never expose API keys in meta tags, JSON-LD, or llms.txt. Keep crawlable content free of internal URLs and tokens.
• Run it from WSL with Lighthouse + the chrome-devtools MCP to audit SEO locally before shipping.

Official docs:

• https://developers.google.com/search/docs
• https://developer.chrome.com/docs/lighthouse/seo
• https://www.sitemaps.org/protocol.html
• https://llmstxt.org/
• https://schema.org/
• https://web.dev/articles/vitals