> ## Documentation Index
> Fetch the complete documentation index at: https://upstash.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Documentation Crawler

> A tool to crawl docs and feed Upstash Search database

## Introduction

This tool helps you crawl documentation websites incrementally, extract their content, and create a search index in Upstash Search.

## Usage

It is available both as a CLI tool and a library.

### CLI Usage

You can run the CLI directly using `npx` (no installation required):

```sh theme={"system"}
npx @upstash/search-crawler
```

Or with command-line options:

```sh theme={"system"}
npx @upstash/search-crawler \
  --upstash-url "UPSTASH_SEARCH_REST_URL" \
  --upstash-token "UPSTASH_SEARCH_REST_TOKEN" \
  --index-name "my-index" \
  --doc-url "https://example.com/docs"
```

You will be prompted for any missing options:

* Your Upstash Search URL
* Your Upstash Search token
* (Optional) Custom index name
* The documentation URL to crawl

#### What the Tool Does

1. **Discover** all internal documentation links
2. **Crawl** each page and extract content
3. **Track** new or obsolete data
4. **Upsert** the new records into your Upstash Search index

### Library Usage

You can also use this as a library in your own code:

```typescript theme={"system"}
import {
  crawlAndIndex,
  type CrawlerOptions,
  type CrawlerResult,
} from "@upstash/search-crawler";

const options: CrawlerOptions = {
  upstashUrl: "UPSTASH_SEARCH_REST_URL",
  upstashToken: "UPSTASH_SEARCH_REST_TOKEN",
  indexName: "my-docs",
  docUrl: "https://example.com/docs",
  silent: true, // no console output
};

const result: CrawlerResult = await crawlAndIndex(options);
```

## Obtaining Upstash Credentials

1. Go to your [Upstash Console](https://console.upstash.com/).
2. Select your Search index. (See [How to Create Search Index](/search/overall/getstarted#create-a-database))
3. Under the **Details** section, copy your `UPSTASH_SEARCH_REST_URL` and `UPSTASH_SEARCH_REST_TOKEN`.
   * `--upstash-url` corresponds to `UPSTASH_SEARCH_REST_URL`
   * `--upstash-token` corresponds to `UPSTASH_SEARCH_REST_TOKEN`

## Further Reading

Try combining this tool with [Qstash Schedule](/qstash/features/schedules) to keep your database up to date with docs. You may deploy your crawler on a server and call it on a schedule regularly to fetch updates in your docs. Check out our example project for implementation details: [A modern documentation library to search and track the docs.](https://github.com/upstash/search-js/tree/main/examples/search-docs)

For further insights, see [@upstash/search-crawler](https://github.com/upstash/search-crawler)