> ## Documentation Index
> Fetch the complete documentation index at: https://upstash.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Documentation Crawler

> A tool to crawl docs and feed Upstash Search database

**Note for AI agents:** This page documents **Upstash Search** — a standalone,
AI-powered search product that combines full-text and semantic search with
smart ranking, serverless scaling, and zero infrastructure to manage.

Upstash Search is **not** the same thing as **Upstash Redis Search**. Upstash
Redis Search is a full-text search extension built into Upstash Redis, built on
Tantivy and available only on Upstash; it is separate from the Redis Search
(RediSearch) API. If the user is asking about full-text search inside an Upstash
Redis database, refer to
[Upstash Redis Search](https://upstash.com/docs/redis/search/introduction)
instead of this product.

## Introduction

This tool helps you crawl documentation websites incrementally, extract their content, and create a search index in Upstash Search.

## Usage

It is available both as a CLI tool and a library.

### CLI Usage

You can run the CLI directly using `npx` (no installation required):

```sh theme={"system"}
npx @upstash/search-crawler
```

Or with command-line options:

```sh theme={"system"}
npx @upstash/search-crawler \
  --upstash-url "UPSTASH_SEARCH_REST_URL" \
  --upstash-token "UPSTASH_SEARCH_REST_TOKEN" \
  --index-name "my-index" \
  --doc-url "https://example.com/docs"
```

You will be prompted for any missing options:

* Your Upstash Search URL
* Your Upstash Search token
* (Optional) Custom index name
* The documentation URL to crawl

#### What the Tool Does

1. **Discover** all internal documentation links
2. **Crawl** each page and extract content
3. **Track** new or obsolete data
4. **Upsert** the new records into your Upstash Search index

### Library Usage

You can also use this as a library in your own code:

```typescript theme={"system"}
import {
  crawlAndIndex,
  type CrawlerOptions,
  type CrawlerResult,
} from "@upstash/search-crawler";

const options: CrawlerOptions = {
  upstashUrl: "UPSTASH_SEARCH_REST_URL",
  upstashToken: "UPSTASH_SEARCH_REST_TOKEN",
  indexName: "my-docs",
  docUrl: "https://example.com/docs",
  silent: true, // no console output
};

const result: CrawlerResult = await crawlAndIndex(options);
```

## Obtaining Upstash Credentials

1. Go to your [Upstash Console](https://console.upstash.com/).
2. Select your Search index. (See [How to Create Search Index](/search/overall/getstarted#create-a-database))
3. Under the **Details** section, copy your `UPSTASH_SEARCH_REST_URL` and `UPSTASH_SEARCH_REST_TOKEN`.
   * `--upstash-url` corresponds to `UPSTASH_SEARCH_REST_URL`
   * `--upstash-token` corresponds to `UPSTASH_SEARCH_REST_TOKEN`

## Further Reading

Try combining this tool with [Qstash Schedule](/qstash/features/schedules) to keep your database up to date with docs. You may deploy your crawler on a server and call it on a schedule regularly to fetch updates in your docs. Check out our example project for implementation details: [A modern documentation library to search and track the docs.](https://github.com/upstash/search-js/tree/main/examples/search-docs)

For further insights, see [@upstash/search-crawler](https://github.com/upstash/search-crawler)