Documentation Crawler - Upstash Documentation

Introduction

This tool helps you crawl documentation websites incrementally, extract their content, and create a search index in Upstash Search.

Usage

It is available both as a CLI tool and a library.

CLI Usage

You can run the CLI directly using npx (no installation required):

npx @upstash/search-crawler

Or with command-line options:

npx @upstash/search-crawler \
  --upstash-url "UPSTASH_SEARCH_REST_URL" \
  --upstash-token "UPSTASH_SEARCH_REST_TOKEN" \
  --index-name "my-index" \
  --doc-url "https://example.com/docs"

You will be prompted for any missing options:

Your Upstash Search URL
Your Upstash Search token
(Optional) Custom index name
The documentation URL to crawl

What the Tool Does

Discover all internal documentation links
Crawl each page and extract content
Track new or obsolete data
Upsert the new records into your Upstash Search index

Library Usage

You can also use this as a library in your own code:

import {
  crawlAndIndex,
  type CrawlerOptions,
  type CrawlerResult,
} from "@upstash/search-crawler";

const options: CrawlerOptions = {
  upstashUrl: "UPSTASH_SEARCH_REST_URL",
  upstashToken: "UPSTASH_SEARCH_REST_TOKEN",
  indexName: "my-docs",
  docUrl: "https://example.com/docs",
  silent: true, // no console output
};

const result: CrawlerResult = await crawlAndIndex(options);

Obtaining Upstash Credentials

Go to your Upstash Console.
Select your Search index. (See How to Create Search Index)
Under the Details section, copy your UPSTASH_SEARCH_REST_URL and UPSTASH_SEARCH_REST_TOKEN.
- --upstash-url corresponds to UPSTASH_SEARCH_REST_URL
- --upstash-token corresponds to UPSTASH_SEARCH_REST_TOKEN

​Introduction

​Usage

​CLI Usage

​What the Tool Does

​Library Usage

​Obtaining Upstash Credentials

​Further Reading