# Sigmie — Full Documentation

> A modern, developer-friendly Elasticsearch and OpenSearch library for PHP and Laravel. Fluent search, semantic and hybrid retrieval, AI-ready, no boilerplate.

This file concatenates the full Sigmie documentation for one-shot LLM ingestion. Individual pages live under /docs/v2/{slug}.md.



---

<!-- source: https://sigmie.com/docs/v2/introduction -->

# Introduction

Sigmie is a PHP library for Elasticsearch and OpenSearch. You describe what a field holds and what the search should do. Sigmie writes the mappings, analyzers, and query DSL for you.

```php
$results = $sigmie->newSearch('products')
    ->properties($props)
    ->queryString('lapto')
    ->typoTolerance()
    ->typoTolerantAttributes(['name'])
    ->filters('price<=1500 AND in_stock:true')
    ->facets('brand price:100')
    ->highlighting(['name', 'description'], prefix: '<mark>', suffix: '</mark>')
    ->sort('_score:desc price:asc')
    ->get();
```

Written by hand, the same request is a nested array of `bool`, `match`, `range`, `highlight`, and aggregation clauses. Sigmie builds that array from the chain and the field types.

## Why Sigmie exists

Elasticsearch is flexible, and that flexibility costs time. A good product search needs the right analyzer on each field, a keyword sub-field for filters, a sortable sub-field for sorting, fuzzy matching on some fields but not others, and aggregations that ignore the filter of their own facet. Each decision lives in a different part of a large JSON body.

Sigmie moves those decisions into three places that PHP code can read:

| Place | You write | Sigmie generates |
|-------|-----------|------------------|
| Field types | `$props->title('name')` | Mapping, analyzer, sub-fields, and the queries for that field |
| Filter and sort strings | `'price:100..500 AND in_stock:true'` | `bool` filter clauses and `sort` entries |
| The search chain | `->typoTolerance()->facets('brand')` | Query clauses, fuzziness, and aggregations |

The trade-off is control. Sigmie picks the analyzer and query shape for each field type. When the defaults do not fit, `newQuery()` builds a query clause by clause, and `rawQuery()` sends a query array as is. See [Advanced queries](https://sigmie.com/docs/v2/query.md).

## Field types carry the search behavior

Elasticsearch knows `text`, `keyword`, and number types. Sigmie adds types that say what the data is for:

```php
$props = new NewProperties;
$props->title('name');           // word-boundary tokens, phrase-prefix matching, sortable
$props->category('brand');       // full text plus exact filters and facets
$props->price();                 // scaled_float with range filters and histogram facets
$props->text('bio')->keyword();  // full text plus a keyword sub-field for exact filters
```

The type decides the mapping and the queries. A `title()` field runs a phrase-prefix query, so `lapto` already matches "Laptop Pro". A `price()` field returns min, max, and a histogram when you facet on it. See [Mappings and properties](https://sigmie.com/docs/v2/mappings.md) for every type.

## Filter strings replace boolean queries

A filter is a string that the [filter parser](https://sigmie.com/docs/v2/filter-parser.md) turns into `bool` filter clauses:

```php
->filters('category:"electronics" AND price:100..500 AND in_stock:true')
```

The parser checks each field against the properties you pass. A filter on a missing or non-filterable field adds a message to the response's `errors` list instead of failing silently. Pass `throwOnError: true` to get a `ParseException` instead. Sort strings work the same way through the [sort parser](https://sigmie.com/docs/v2/sort-parser.md).

## Semantic search uses the same chain

A text field becomes semantic when you call `semantic()` on it and register an embeddings API under the name it uses:

```php
$props->text('description')->semantic(api: 'embeddings', dimensions: 384);

$sigmie->registerApi('embeddings', $embeddingsApi);

$results = $sigmie->newSearch('products')
    ->properties($props)
    ->semantic()
    ->queryString('portable work computer')   // matches "laptop" and "notebook"
    ->get();
```

Sigmie embeds the field when it indexes a document, and embeds the query string when it searches. Keyword matching, filters, and facets still apply. See [Semantic search](https://sigmie.com/docs/v2/semantic-search.md).

## When to use Sigmie

Sigmie fits when search is a feature of your PHP application: a product catalog, a content library, a support knowledge base, or retrieval for an AI agent. It also fits Laravel projects through the [Laravel Scout](https://sigmie.com/docs/v2/laravel-scout.md) driver and the [Laravel AI SDK](https://sigmie.com/docs/v2/laravel-ai.md) tools.

Use the official Elasticsearch client instead when you need a feature Sigmie does not expose, or when a large codebase already builds raw queries. The two can share a cluster. Inside Sigmie, `rawQuery()` sends a query array unchanged.

## Next steps

- [Installation](https://sigmie.com/docs/v2/installation.md) installs the package and connects to a cluster.
- [Quick start](https://sigmie.com/docs/v2/quick-start.md) builds a product search from an empty index.
- [Core concepts](https://sigmie.com/docs/v2/core-concepts.md) explains the client, indices, documents, and properties.


---

<!-- source: https://sigmie.com/docs/v2/installation -->

# Installation

Install the package, point it at a cluster, and confirm that the connection works.

## Check the requirements

- PHP 8.1 or higher
- Composer
- Elasticsearch 7.x, 8.x, or 9.x, or OpenSearch 2.x or 3.x

## Install the package

```bash
composer require sigmie/sigmie
```

Outside a framework, load the Composer autoloader once in your entry script:

```php
require_once 'vendor/autoload.php';
```

## Start Elasticsearch locally

Skip this step if you already have a cluster. To run a single node with security off, use Docker:

```bash
docker run -d \
  --name elasticsearch \
  -p 9200:9200 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  -e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
  docker.elastic.co/elasticsearch/elasticsearch:9.1.3
```

When the container is up, check the node:

```bash
curl http://localhost:9200
```

The response is a JSON document that ends with `"tagline" : "You Know, for Search"`.

For a local stack with embedding and reranking services, see [Docker](https://sigmie.com/docs/v2/docker.md).

## Connect to the cluster

Pass the host to `Sigmie::create()`:

```php
use Sigmie\Sigmie;

$sigmie = Sigmie::create(hosts: ['127.0.0.1:9200']);
```

That is enough for a local node with security off. For other clusters, pick the case that matches yours.

### Connect to a secured cluster

Pass credentials in the `config` array:

```php
$sigmie = Sigmie::create(
    hosts: ['https://elasticsearch.example.com:9200'],
    config: [
        'auth' => ['elastic', 'your-password'],
        'verify' => true,
    ]
);
```

The `config` array accepts any [Guzzle request option](https://docs.guzzlephp.org/en/stable/request-options.html). For API keys, bearer tokens, and custom CA files, see [Connection setup](https://sigmie.com/docs/v2/connection.md).

### Connect to OpenSearch

Set the `engine` argument:

```php
use Sigmie\Enums\SearchEngineType;

$sigmie = Sigmie::create(
    hosts: ['https://localhost:9200'],
    engine: SearchEngineType::OpenSearch,
    config: [
        'auth' => ['admin', 'MyStrongPass123!'],
        'verify' => false,   // self-signed cert
    ]
);
```

See [OpenSearch](https://sigmie.com/docs/v2/opensearch.md) for the differences and for AWS OpenSearch Service.

### Connect to several nodes

List every node:

```php
$sigmie = Sigmie::create(
    hosts: ['10.0.0.1:9200', '10.0.0.2:9200', '10.0.0.3:9200']
);
```

Sigmie sends requests to the nodes in turn. If a node fails with a network error, that request fails and later requests skip the node. Sigmie does not retry the failed request on another node.

## Verify the connection

```php
var_dump($sigmie->isConnected());
```

The output is `bool(true)`. If it is `bool(false)`, see the next section. `isConnected()` returns `false` on any error and does not throw.

Continue with the [Quick start](https://sigmie.com/docs/v2/quick-start.md) to build your first search.

## Fix connection errors

Errors 7 and 28 reach your code as `Elastic\Transport\Exception\NoNodeAvailableException`, with the cURL error in `$e->getPrevious()`. Error 60 reaches your code as a Guzzle `RequestException`.

| Error | Cause | Fix |
|-------|-------|-----|
| `cURL error 7: Failed to connect` | Elasticsearch is not running, or the host or port is wrong. | Start the node, or correct `hosts`. |
| `cURL error 28: Operation timed out` | The node did not answer in time. | Raise `connect_timeout` (default 15 seconds) and `timeout` in `config`. |
| `cURL error 60: SSL certificate problem` | PHP does not trust the cluster certificate. | In production, set `verify` to the path of your CA bundle. In development only, set `'verify' => false`. |
| `Sigmie\Base\ElasticsearchException` with code 401 | The credentials are wrong, or the cluster expects a different auth method. | Check `auth` and the cluster security settings. |

For every connection option, see [Connection setup](https://sigmie.com/docs/v2/connection.md).


---

<!-- source: https://sigmie.com/docs/v2/quick-start -->

# Quick start

In this tutorial, we build a search for a catalog of three products. By the end, it finds a product from a misspelled word, filters by stock and price, sorts by price, and counts products per category.

All code goes in one file, `search.php`. Each step adds code to the end of the file, and you run the whole file again.

## Before you start

You need:

- Sigmie in a Composer project. See [Installation](https://sigmie.com/docs/v2/installation.md).
- Elasticsearch on `127.0.0.1:9200` with security off. The Docker command in [Start Elasticsearch locally](https://sigmie.com/docs/v2/installation.md#start-elasticsearch-locally) gives you one.

## Step 1: Connect

Create `search.php` in your project root:

```php
<?php

require_once 'vendor/autoload.php';

use Sigmie\Document\Document;
use Sigmie\Mappings\NewProperties;
use Sigmie\Sigmie;

$sigmie = Sigmie::create(hosts: ['127.0.0.1:9200']);

var_dump($sigmie->isConnected());
```

Run it:

```bash
php search.php
```

The output is:

```
bool(true)
```

If you see `bool(false)`, Elasticsearch is not reachable. Check that the node runs on port 9200.

## Step 2: Create the index and add products

First, we describe the fields. `NewProperties` holds the field types, and each type decides how Elasticsearch stores and searches the field. Add this to `search.php`:

```php
$props = new NewProperties;
$props->title('name');
$props->text('description');
$props->category('category');
$props->price();
$props->bool('in_stock');
```

Now, we create the `products` index from those properties and add three products. `deleteIfExists()` removes the index from an earlier run, so you can run the script again:

```php
$sigmie->deleteIfExists('products');

$sigmie->newIndex('products')
    ->properties($props)
    ->create();

$sigmie->collect('products', refresh: true)
    ->properties($props)
    ->merge([
        new Document([
            'name' => 'Laptop Pro',
            'description' => 'High-performance laptop for professionals',
            'category' => 'electronics',
            'price' => 1299,
            'in_stock' => true,
        ]),
        new Document([
            'name' => 'Wireless Mouse',
            'description' => 'Ergonomic wireless mouse with precision tracking',
            'category' => 'accessories',
            'price' => 49,
            'in_stock' => true,
        ]),
        new Document([
            'name' => 'USB-C Cable',
            'description' => 'Fast charging and data transfer cable',
            'category' => 'accessories',
            'price' => 15,
            'in_stock' => false,
        ]),
    ]);

echo $sigmie->collect('products')->count(), "\n";
```

`refresh: true` makes the products searchable as soon as `merge()` returns. Leave it out for large imports. See [Documents](https://sigmie.com/docs/v2/document.md).

Run the script. The output now ends with:

```
3
```

## Step 3: Search for a word

We print product names in every step from here on. Add a small helper and the first search:

```php
function printNames($results): void
{
    foreach ($results->hits() as $hit) {
        echo $hit['name'], "\n";
    }
}

echo "-- laptop\n";

printNames($sigmie->newSearch('products')
    ->properties($props)
    ->queryString('laptop')
    ->get());
```

Run the script. The new output is:

```
-- laptop
Laptop Pro
```

## Step 4: Find a misspelled word

Users mistype. Search for `laptp`, with typo tolerance on the `name` field:

```php
echo "-- laptp\n";

printNames($sigmie->newSearch('products')
    ->properties($props)
    ->queryString('laptp')
    ->typoTolerance()
    ->typoTolerantAttributes(['name'])
    ->get());
```

Run the script. The new output is:

```
-- laptp
Laptop Pro
```

Without the two highlighted lines, this search finds nothing. `typoTolerance()` sets the rules: one typo in words of 3 or more characters, two typos in words of 6 or more. `typoTolerantAttributes()` names the fields the rules apply to.

## Step 5: Filter by stock and price

A filter string limits the results without changing their order. An empty query string matches every product:

```php
echo "-- in stock, 100 or less\n";

printNames($sigmie->newSearch('products')
    ->properties($props)
    ->queryString('')
    ->filters('in_stock:true AND price<=100')
    ->get());
```

Run the script. The new output is:

```
-- in stock, 100 or less
Wireless Mouse
```

The cable costs less than 100, but it is out of stock. The [filter parser](https://sigmie.com/docs/v2/filter-parser.md) page lists every operator.

## Step 6: Sort by price

```php
echo "-- cheapest first\n";

printNames($sigmie->newSearch('products')
    ->properties($props)
    ->queryString('')
    ->sort('price:asc')
    ->get());
```

Run the script. The new output is:

```
-- cheapest first
USB-C Cable
Wireless Mouse
Laptop Pro
```

Without `sort()`, Sigmie sorts by `_score`, highest first. Sigmie rejects `_score:asc`. The [sort parser](https://sigmie.com/docs/v2/sort-parser.md) page lists the sort syntax.

## Step 7: Count products per category

Facets return counts for a filter sidebar. Ask for category counts and the price range:

```php
echo "-- facets\n";

$results = $sigmie->newSearch('products')
    ->properties($props)
    ->queryString('')
    ->facets('category price:100')
    ->get();

$facets = $results->json('facets');

print_r((array) $facets['category']);

$price = (array) $facets['price'];

echo "Price: {$price['min']} to {$price['max']}\n";
```

Run the script. The new output is:

```
-- facets
Array
(
    [accessories] => 2
    [electronics] => 1
)
Price: 15 to 1299
```

`price:100` sets a histogram interval of 100. The price facet also holds that histogram under `histogram`.

## What we built

`search.php` now creates an index, adds three products, and runs a word search, a typo-tolerant search, a filtered search, a sorted search, and a facet count. Every step used the same `newSearch()` chain, with one more method each time.

## Next steps

- [Search](https://sigmie.com/docs/v2/search.md) covers field weights, highlighting, pagination, and every other `newSearch()` method.
- [Facets](https://sigmie.com/docs/v2/facets.md) explains facets that keep counts for other values after a user selects one.
- [Semantic search](https://sigmie.com/docs/v2/semantic-search.md) finds products by meaning with an embeddings API.
- [Mappings and properties](https://sigmie.com/docs/v2/mappings.md) lists every field type.
- [Laravel Scout](https://sigmie.com/docs/v2/laravel-scout.md) connects Sigmie to Eloquent models.


---

<!-- source: https://sigmie.com/docs/v2/core-concepts -->

# Core Concepts

Sigmie has four parts. The client holds the connection. An index stores documents. A document is one JSON record. Properties describe each field, and Sigmie reads them in three places:

```
                     ┌──► newIndex()->properties()    mapping and analyzers
NewProperties $props ┼──► collect()->properties()     validation and embeddings
                     └──► newSearch()->properties()   the queries for each field
```

The rest of this page explains each part and why Sigmie wires them this way. For a first working search, follow the [Quick Start](https://sigmie.com/docs/v2/quick-start.md).

## The client is the entry point

```php
use Sigmie\Sigmie;

$sigmie = Sigmie::create(hosts: ['127.0.0.1:9200']);
```

Every other builder starts from the client:

```php
$sigmie->newIndex('movies');                // build a new index
$sigmie->index('movies');                   // load an existing one
$sigmie->collect('movies');                 // read and write documents
$sigmie->newSearch('movies');               // build a user-facing search
$sigmie->newQuery('movies');                // build a raw query
$sigmie->newRecommend('movies');            // build a recommendation
```

[Installation](https://sigmie.com/docs/v2/installation.md) and [Connection Setup](https://sigmie.com/docs/v2/connection.md) cover hosts, credentials, and OpenSearch.

## An index name is an alias

An index holds related documents, like a table in a database. When you call `$sigmie->newIndex('movies')->create()`, Sigmie creates a physical index with a timestamp suffix and points the alias `movies` at it:

```
movies (alias) ──► movies_20260101120000000000 (physical index)
```

Your code only uses `movies`. The alias exists because Elasticsearch cannot change the analysis or the field types of an index that already holds documents. To change them, Sigmie builds a new physical index, fills it, and then moves the alias to it. Searches keep using the old index until the move. [Update an index](https://sigmie.com/docs/v2/index.md#update-an-index) and [Rebuild an index](https://sigmie.com/docs/v2/index.md#rebuild-an-index) use this.

### Shards split an index, replicas copy it

A shard is a smaller index that holds part of the documents. An index with 3 shards and 8 documents spreads them, for example, like this:

```
movies
├─ shard 1: document 1, document 2, document 3
├─ shard 2: document 4, document 5, document 6
└─ shard 3: document 7, document 8
```

A replica is a copy of a shard on another node. With 3 primary shards and 2 replicas on 3 nodes, every node holds one primary and a copy of the other two:

```
cluster
├─ node 1: primary 1, replica of 2, replica of 3
├─ node 2: primary 2, replica of 1, replica of 3
└─ node 3: primary 3, replica of 1, replica of 2
```

If a node fails, the other nodes still hold every document, and Elasticsearch promotes a replica to primary. Sigmie creates 1 shard and 2 replicas unless you call `shards()` and `replicas()`. For most workloads, keep each shard under 30 GB.

## Documents go through a collection

A `Document` is a PHP wrapper around one JSON object and its optional `_id`. You write and read documents through a collection, which `$sigmie->collect('movies')` returns:

```php
use Sigmie\Document\Document;

$sigmie->collect('movies')->merge([
    new Document(['title' => 'The Matrix', 'year' => 1999]),
    new Document(['title' => 'Inception', 'year' => 2010]),
]);
```

The collection does not hold the documents in memory. Each call goes to Elasticsearch, so `count()` and `each()` always reflect the index. Elasticsearch makes a write searchable after the next refresh, about one second later by default. [Documents](https://sigmie.com/docs/v2/document.md) covers writing, reading, and refresh control.

## Properties do three jobs

`NewProperties` describes your fields once. Sigmie reads that description at three points:

| Where you pass `$props` | What Sigmie does with it |
|-------------------------|--------------------------|
| `newIndex()->properties()` | Writes the Elasticsearch mapping and the analyzer for each field. |
| `collect()->properties()` | Checks each document before the write, formats `DateTime` values, and generates embeddings for semantic fields. |
| `newSearch()->properties()` | Picks the fields to search and the queries to run on each one. |

Pass the same instance at all three points. A search without properties knows no fields, so a query string returns no hits.

```php
use Sigmie\Mappings\NewProperties;

$props = new NewProperties;
$props->title('title');
$props->name('director');

$sigmie->newIndex('movies')->properties($props)->create();

$sigmie->newSearch('movies')->properties($props)->queryString('matrix')->get();
```

### High-level types carry search behavior

Elasticsearch knows storage types such as `text`, `keyword`, and `integer`. A storage type says nothing about how users search the field. Sigmie adds high-level types that bundle the storage type with an analyzer, sub-fields, and queries. For example, `$props->title('title')` produces:

```
title()  ──► mapping   text field, word tokens, lowercase
         ──► sub-fields title.keyword, title.sortable
         ──► queries   match_phrase_prefix + match
```

A `name()` field uses n-grams instead, so "Wacho" finds "Wachowski". Choose the type that matches how users type into the search box. [Mappings & Properties](https://sigmie.com/docs/v2/mappings.md) lists every type with its mapping and its queries.

## Analysis turns text into tokens

Elasticsearch does not search the stored text. It searches tokens that an analyzer produced at index time. At search time, the same analyzer runs on the query string, so both sides compare equal tokens:

```
Document text: "The Matrix"            Query string: "Matrix"
        │                                      │
   tokenizer ──► ["The", "Matrix"]        tokenizer ──► ["Matrix"]
        │                                      │
   lowercase ──► ["the", "matrix"]        lowercase ──► ["matrix"]
        │                                      │
        └──────────── "matrix" = "matrix" ─────┘   match
```

You set the default analyzer on the index builder, and each high-level type can bring its own:

```php
$sigmie->newIndex('movies')
    ->tokenizeOnWhitespaces()
    ->lowercase()
    ->create();

$sigmie->index('movies')->analyze('The Matrix');   // ["the", "matrix"]
```

[Text Analysis](https://sigmie.com/docs/v2/analysis.md) explains the pipeline. [Tokenizers](https://sigmie.com/docs/v2/tokenizers.md) and [Token Filters](https://sigmie.com/docs/v2/token-filters.md) list the parts.

## Search and query serve different users

Sigmie has two retrieval builders. `newSearch()` takes what a user typed and builds the Elasticsearch request from your properties. It adds typo tolerance, highlighting, and facets on request. `newQuery()` takes the boolean query you write, clause by clause, and sends it as written.

```php
$sigmie->newSearch('movies')
    ->properties($props)
    ->queryString('matrix sci-fi')
    ->typoTolerance()
    ->highlighting(['title'], '<em>', '</em>')
    ->facets('genre')
    ->get();

$sigmie->newQuery('movies')
    ->properties($props)
    ->bool(function ($bool) {
        $bool->must()->match('title', 'matrix');
        $bool->filter()->range('year', ['>' => 1990]);
        $bool->should()->term('genre', 'sci-fi');
    })
    ->get();
```

Use [`newSearch()`](https://sigmie.com/docs/v2/search.md) for a search box. Use [`newQuery()`](https://sigmie.com/docs/v2/query.md) when your code, not a user, decides the exact clauses.


---

<!-- source: https://sigmie.com/docs/v2/index -->

# Indices

These guides cover the life of an index, from creation to deletion. Each index name you pass is an alias over a physical index. [Core Concepts](https://sigmie.com/docs/v2/core-concepts.md#an-index-name-is-an-alias) explains why.

## Create an index

Pass your [properties](https://sigmie.com/docs/v2/mappings.md) to `newIndex()`, then call `create()`:

```php
use Sigmie\Mappings\NewProperties;

$props = new NewProperties;
$props->title('title');
$props->name('director');
$props->category('genre');
$props->number('year')->integer();
$props->date('release_date');

$sigmie->newIndex('movies')
    ->properties($props)
    ->create();
```

`create()` returns the new index. It throws `Sigmie\Index\Alias\AliasAlreadyExists` when the name is taken. To get the existing index instead, call `createIfNotExists()`.

Without `properties()`, Elasticsearch guesses each field type from the first document it receives.

To write documents into the new index, see [Documents](https://sigmie.com/docs/v2/document.md).

## Set the default analyzer

The default analyzer processes every text field that has no analyzer of its own. Chain tokenizer and filter calls before `create()`:

```php
$sigmie->newIndex('movies')
    ->properties($props)
    ->tokenizeOnWhitespaces()
    ->lowercase()
    ->trim()
    ->create();
```

For the available calls, see [Tokenizers](https://sigmie.com/docs/v2/tokenizers.md), [Token Filters](https://sigmie.com/docs/v2/token-filters.md), and [Char Filters](https://sigmie.com/docs/v2/char-filters.md).

### Use a language analyzer

Call `language()` first. It returns a language builder with extra filters, such as the English stemmer:

```php
use Sigmie\Languages\English\English;

$sigmie->newIndex('articles')
    ->language(new English)
    ->properties($props)
    ->englishStemmer()
    ->englishStopwords()
    ->englishLowercase()
    ->create();
```

> **Warning:** `language()` starts a new builder. It keeps the shards and replicas, but drops properties, filters, and `config()` values set before it. Call those after `language()`.

[Languages](https://sigmie.com/docs/v2/language.md) lists the English, German, and Greek builders.

## Set shards and replicas

```php
$sigmie->newIndex('movies')
    ->shards(3)
    ->replicas(1)
    ->create();
```

Without these calls, Sigmie creates 1 shard and 2 replicas. On Elasticsearch Serverless, Sigmie sends neither setting, because the service manages them. [Core Concepts](https://sigmie.com/docs/v2/core-concepts.md#shards-split-an-index-replicas-copy-it) explains shards and replicas.

## Set any other index setting

Pass any [Elasticsearch index module setting](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html) to `config()`:

```php
$sigmie->newIndex('movies')
    ->config('index.max_ngram_diff', 3)
    ->create();
```

## Update an index

To change the properties or the analysis of an index that holds documents, call `update()`. The callback receives an `UpdateIndex` builder with the same methods as `newIndex()`:

```php
use Sigmie\Index\UpdateIndex;

$sigmie->index('movies')->update(function (UpdateIndex $update) {
    $update->properties($newProperties);
    $update->lowercase();

    return $update;
});
```

`update()` copies the documents that are already in Elasticsearch:

1. Creates a new physical index from the callback's settings, such as `movies_20260201120000000000`.
2. Blocks writes to the current index and reindexes every document into the new one.
3. Points the `movies` alias at the new index.
4. Deletes the old index.

Searches keep working through every step. Writes to `movies` fail from step 2 until step 4 ends.

> **Warning:** `update()` does not merge settings. It builds the new index only from what the callback declares. Declare again every property, filter, shard count, and replica count that you want to keep.

## Rebuild an index

When the source of truth lives outside Elasticsearch, such as a database or a CSV feed, call `rebuild()`. It fills a new index from your data, then swaps it in:

```php
use Sigmie\Document\AliveCollection;

$sigmie->newIndex('movies')
    ->properties($props)
    ->rebuild(function (AliveCollection $docs) use ($movies) {
        $docs->merge($movies);
    });
```

`rebuild()` does four things:

1. Creates a new physical index with the builder's settings and mappings.
2. Runs your callback to fill it. Searches keep using the current index.
3. Points the `movies` alias at the new index in one `_aliases` request.
4. Deletes the previous index.

```
movies (alias) ──► movies_20260101   (live, searchable)
                   movies_20260201   ◄── callback fills it

movies (alias) ──► movies_20260201   (one _aliases request)
                   movies_20260101   deleted
```

If no alias exists yet, `rebuild()` creates it. The same call works for the first build and for every refresh after it.

If the callback throws, Sigmie deletes the new index, leaves the alias where it was, and throws the exception again.

`rebuild()` also refuses a result with fewer documents than `minDocuments`, which defaults to 1. An empty result usually means a failed import. Sigmie deletes the new index, keeps the live one, and throws `Sigmie\Index\RebuildTooSmall`. Change the limit with `minDocuments`:

```php
$sigmie->newIndex('movies')->rebuild($fill, minDocuments: 1000);
$sigmie->newIndex('movies')->rebuild($fill, minDocuments: 0);   // allow empty
```

> **Note:** The callback's collection has no properties, the same as `$sigmie->collect()`. To check documents or generate embeddings for semantic fields while you fill it, call `$docs->properties($props)` first.

> **Note:** `rebuild()` does not lock. If two processes can rebuild the same alias at once, run them one after the other yourself. Otherwise the slower rebuild fails at the swap and leaves its new index behind.

### Rebuild across processes

When one process cannot write every document in time, split the rebuild into start, fill, and finish. Any process can resume the pending rebuild by its alias, so a queued job needs only the name:

```php
// Once: create the new index. The alias stays on the live index.
$sigmie->newIndex('movies')->properties($props)->startRebuild();

// In each job: write one chunk.
$sigmie->rebuilding('movies')->collect()->merge($chunk);

// Once all jobs succeed: swap the alias and delete the old index.
$sigmie->rebuilding('movies')->finish(minDocuments: 1);

// If a job fails: drop the new index. Searches never saw it.
$sigmie->rebuilding('movies')->abort();
```

`startRebuild()` turns refresh off and sets replicas to zero for fast bulk writes. `finish()` restores both before the swap. `count()` returns the number of documents written so far.

`rebuilding()` returns the newest index that `startRebuild()` created for the alias and that no alias points to yet. It returns `null` when no rebuild is pending. Sigmie stores the rebuild state in the index's `_meta`, so it survives restarts and queue retries. `rebuild()` is `startRebuild()`, your callback, then `finish()`.

## Inspect an index

`$sigmie->index()` returns the index, or `null` when it does not exist:

```php
$index = $sigmie->index('movies');

$index->name;                            // physical name, e.g. movies_20260101120000000000
$index->mappings;                        // index mappings
$index->mappings->properties();          // property definitions
$index->settings;                        // index settings
$index->raw;                             // raw Elasticsearch response
$index->analyze('The Matrix');           // tokens from the default analyzer
```

## Delete an index

```php
$sigmie->index('movies')->delete();
```

To delete without loading the index first, call `$sigmie->delete()` or `$sigmie->deleteIfExists()`. Both take an alias or a physical name, or several names separated by commas:

```php
$sigmie->deleteIfExists('movies,series');
```


---

<!-- source: https://sigmie.com/docs/v2/document -->

# Documents

These guides cover the documents in one index. You work with them through a collection, which `$sigmie->collect()` returns. [Core Concepts](https://sigmie.com/docs/v2/core-concepts.md#documents-go-through-a-collection) explains how a collection relates to the index.

## Create a document

A `Document` wraps one JSON object. It holds any structure that PHP can encode as JSON:

```php
use Sigmie\Document\Document;

$doc = new Document([
    'title' => 'Inception',
    'director' => [
        'name' => 'Christopher Nolan',
        'born' => 1970,
    ],
    'cast' => [
        ['name' => 'Leonardo DiCaprio', 'role' => 'Cobb'],
        ['name' => 'Marion Cotillard', 'role' => 'Mal'],
    ],
]);
```

To choose the `_id` yourself, pass it as the second argument. Without it, Elasticsearch generates one on the first write:

```php
$doc = new Document(['title' => 'The Matrix'], 'matrix_1999');
```

Read and change fields with array access or properties. Array reads also accept dotted paths:

```php
$doc['title'] = 'New Title';
$doc['director.name'];      // 'Christopher Nolan'
$doc->title;                // 'New Title'
unset($doc['cast']);
```

## Get a collection

```php
$movies = $sigmie->collect('movies');
```

To make every write searchable before the call returns, pass `refresh: true`. Use it in tests:

```php
$movies = $sigmie->collect('movies', refresh: true);
```

> **Warning:** Do not use `refresh: true` in production. It forces a refresh on every write, which slows indexing.

## Add documents

To add many documents, call `merge()`. It sends one bulk request:

```php
$movies->merge([
    new Document(['title' => 'Snow White']),
    new Document(['title' => 'Cinderella']),
    new Document(['title' => 'Sleeping Beauty']),
]);
```

To add one document, call `add()`. It returns the document with its `_id` set:

```php
$doc = $movies->add(new Document(['title' => 'Mickey Mouse']));

$doc->_id;   // generated by Elasticsearch
```

`add()` only creates. If a document with the same `_id` exists, it throws `Sigmie\Base\ElasticsearchException`. To overwrite, see [Update a document](#update-a-document).

## Check documents against your properties

Pass your [properties](https://sigmie.com/docs/v2/mappings.md) to the collection. Sigmie then checks each document before it writes the batch:

```php
use Sigmie\Mappings\NewProperties;

$props = new NewProperties;
$props->title('title');
$props->date('release_date');
$props->number('rating')->float();

$movies = $sigmie->collect('movies')->properties($props);

$movies->merge([
    new Document([
        'title' => 'The Matrix',
        'release_date' => '1999-03-31',
        'rating' => 8.7,
    ]),
]);
```

If a value does not fit its property, such as a non-numeric `rating`, Sigmie writes nothing and throws `InvalidArgumentException`. The message starts with `Document validation failed:` and names each bad field. Sigmie checks only the fields that the properties define and skips `null` values.

The collection with properties also converts `DateTime` and `Carbon` values to the string format of the field, and generates embeddings for [semantic fields](https://sigmie.com/docs/v2/semantic-search.md).

## Update a document

Pick the method by what should happen to the fields you leave out:

| Method | Fields you leave out |
|--------|----------------------|
| `replace()` | Removed |
| `merge()` | Kept |

To replace the whole document, call `replace()` with the same `_id`:

```php
$movies->replace(new Document([
    'title' => 'The Matrix',
    'year' => 1999,
], 'matrix_1'));
```

To change some fields and keep the rest, call `merge()` with the same `_id`. If no document has that `_id`, `merge()` creates it:

```php
$movies->merge([
    new Document(['rating' => 8.7], 'matrix_1'),
]);
```

## Read documents

```php
$movies->get('matrix_1');                        // one document, or null
$movies->getMany(['matrix_1', 'inception_1']);   // several, by _id
$movies->has('matrix_1');                        // true when the _id exists
$movies->count();                                // number of documents
$movies->take(10);                               // first 10 documents
$movies->random(5);                              // 5 random documents
$movies->toArray();                              // every document, keyed by _id
```

> **Warning:** `toArray()` loads the whole index into memory. Use it only on small indices. To go through a large index, see [Iterate over every document](#iterate-over-every-document).

To return only some fields, call `only()` or `except()` before `get()`, `getMany()`, `take()`, `random()`, or `each()`. Dotted paths reach nested fields:

```php
$movies->only(['title', 'year'])->random(5);
$customers->except(['contacts.email'])->getMany(['customer_1', 'customer_2']);
```

## Iterate over every document

`each()` visits every document in the index without loading the index into memory:

```php
$movies->each(function (Document $doc): void {
    echo $doc['title'] . "\n";
});
```

Sigmie reads 500 documents per request. To change the page size, call `chunk()`:

```php
$movies->chunk(100)->each(function (Document $doc): void {
    processOne($doc);
});
```

Sigmie pages with a point in time and `search_after`. The loop sees the index as it was when the loop started, so writes during the loop do not shift the pages.

To iterate over only the documents that match a search, use `each()` or `lazy()` on `newSearch()`. See [Iterating over all matching hits](https://sigmie.com/docs/v2/search.md#iterating-over-all-matching-hits).

## Delete documents

```php
$movies->remove('matrix_1');                     // one document, by _id
$movies->remove(['matrix_1', 'inception_1']);    // several, in one bulk request
$movies->clear();                                // every document; the index stays
```

To delete the index itself, see [Delete an index](https://sigmie.com/docs/v2/index.md#delete-an-index).

## Write values for typed fields

Some property types expect a specific value shape. [Mappings & Properties](https://sigmie.com/docs/v2/mappings.md) lists the rules for each type.

A `date()` field takes a `Y-m-d` string. A `datetime()` field takes an ISO 8601 timestamp:

```php
$props->date('release_date');
$props->datetime('created_at');

new Document([
    'release_date' => '2023-04-07',
    'created_at' => '2023-04-07T12:38:29.000000Z',
]);
```

A `geoPoint()` field takes `lat` and `lon` keys:

```php
$props->geoPoint('location');

new Document([
    'venue' => 'Cinema Downtown',
    'location' => ['lat' => 40.7128, 'lon' => -74.0060],
]);
```

A `nested()` field takes a list of objects:

```php
$props->nested('cast', function (NewProperties $props) {
    $props->name('actor');
    $props->keyword('role');
});

new Document([
    'title' => 'Avengers',
    'cast' => [
        ['actor' => 'Robert Downey Jr.', 'role' => 'Iron Man'],
        ['actor' => 'Chris Evans', 'role' => 'Captain America'],
    ],
]);
```

To filter on the items of a nested field, see [Filter Parser](https://sigmie.com/docs/v2/filter-parser.md#nested-field-filtering).

## Make writes visible to search

Elasticsearch makes a write searchable at the next refresh, about one second later by default. Until then, searches and `count()` do not see it:

```php
$movies = $sigmie->collect('movies');
$movies->add(new Document(['title' => 'Snow White']));
$movies->count();   // 0
sleep(1);
$movies->count();   // 1
```

For a batch import, write with the default and refresh once at the end:

```php
$movies = $sigmie->collect('movies');
$movies->merge($manyDocuments);
$movies->refresh();
```

`$sigmie->refresh('movies')` does the same without a collection.


---

<!-- source: https://sigmie.com/docs/v2/mappings -->

# Mappings & Properties

`NewProperties` builds the schema of an index. Each method adds one field and returns it, so you can chain modifiers onto it:

```php
use Sigmie\Mappings\NewProperties;

$props = new NewProperties;
$props->title('title');
$props->name('director');
$props->number('year')->integer();
```

Pass the same instance to `newIndex()`, `collect()`, and `newSearch()`. [Core Concepts](https://sigmie.com/docs/v2/core-concepts.md#properties-do-three-jobs) explains what Sigmie does with it at each point.

The type tables below use these columns:

- Default name is the field name when you call the method without one.
- Mapping is the Elasticsearch field type, its analysis, and its sub-fields.
- Queries are the queries `newSearch()` runs on the field for a query string.
- Accepts lists the values that pass the check in a collection with properties. Sigmie checks each item of a list on its own.

## High-level types

High-level types bundle a mapping with the queries that suit one kind of data.

| Method | Default name | Mapping | Queries | Accepts |
|--------|--------------|---------|---------|---------|
| `title()` | `title` | `text`, word tokens, lowercase. Sub-fields `.keyword`, `.sortable`. | `match_phrase_prefix`, `match` | string |
| `name()` | `name` | `text`, 4 to 5 character n-grams, index prefixes. Sub-fields `.keyword`, `.sortable`, and `.{name}_text` for prefixes. | `match` on n-grams, `match_phrase_prefix` and `match_bool_prefix` on `.{name}_text` | string |
| `category()` | `category` | `text`. Sub-fields `.keyword`, `.sortable`. | `match`, `term` on `.keyword`, `match_phrase_prefix` | string |
| `tags()` | `tags` | `text`, `#` removed, word tokens, lowercase. | `match`, `match_phrase_prefix` | string |
| `price()` | `price` | `scaled_float`, scaling factor 100. | none | number |
| `longText()` | none | `text`, no sub-fields. | `match` | string |
| `shortText()` | none | Same as `longText()`. | `match` | string |
| `html()` | none | `text`, HTML tags stripped, word tokens, trimmed, lowercase, repeated tokens removed. | `match` | string |
| `searchableNumber()` | none | `text`, every non-digit removed. Sub-fields `.keyword`, `.sortable`. | `match`, `match_phrase_prefix` | string |
| `id()` | none | `keyword`, case-sensitive. Sub-field `.sortable` (integer). | `term`, `prefix` | integer |
| `email()` | `email` | `text`, split on `@` and `.`, lowercase, index prefixes. Sub-fields `.keyword`, `.sortable`. | `match`, `prefix`, `term` on `.keyword` | string |
| `address()` | `address` | `text`, word tokens, lowercase, index prefixes. Sub-fields `.keyword`, `.sortable`. | `match_bool_prefix` | string |
| `caseSensitiveKeyword()` | none | `keyword`, no lowercase normalizer. | `term`, `prefix` | string |
| `path()` | none | `text`, one token per level (`/a`, `/a/b`, `/a/b/c`), lowercase, index prefixes. Sub-field `.keyword`. | `prefix`, `match` | string |
| `boost()` | `boost` | `float`. | none | number |

```php
$props->title('title');
$props->name('director');
$props->category('genre');
$props->tags('skills');
$props->price();                       // field name 'price'
$props->longText('synopsis');
$props->html('content');
$props->searchableNumber('phone');
$props->id('user_id');
$props->email('contact_email');
$props->caseSensitiveKeyword('promo_code');
$props->path('file_path');
$props->boost();
```

`boost()` multiplies the score of each hit by the document's value of the field. A document without a value keeps its score.

`searchableNumber()` stores the number as text, so users find it by typing it. To filter, sort, or aggregate numbers, use `number()`.

### Facet logic

Every type that supports facets, such as `category()`, `keyword()`, and `price()`, takes a facet logic. The default is conjunctive:

```php
$props->category('brand')->facetDisjunctive();   // OR between selected values
$props->category('color')->facetConjunctive();   // AND between selected values
```

[Facets](https://sigmie.com/docs/v2/facets.md) explains both.

## Native types

Native types map one to one to an Elasticsearch field type.

| Method | Mapping | Queries | Accepts |
|--------|---------|---------|---------|
| `text()` | `text` with the index default analyzer. | `match` | string |
| `keyword()` | `keyword` with a lowercase normalizer. | `term`, `prefix` | string |
| `number()` | `integer` until you call a type modifier. | none | number |
| `integer()`, `float()`, `long()`, `double()`, `scaledFloat()` | Shortcuts for `number()` with that type. | none | number |
| `bool()` | `boolean` | `term` for `true` when the query string equals the field name | `true` or `false` |
| `date()` | `date`, format `strict_date_optional_time_nanos`. | none | `Y-m-d` string |
| `datetime()` | `date`, format `strict_date_optional_time_nanos`. | none | timestamp string, see [Date and datetime](#date-and-datetime) |
| `geoPoint()` | `geo_point` | none | `['lat' => …, 'lon' => …]`, or a list of them |
| `range()` | `integer_range` until you call a type modifier. | none | array with at least one of `gte`, `gt`, `lte`, `lt` |

To filter by distance from a `geoPoint()` field, see [Geo-location filtering](https://sigmie.com/docs/v2/filter-parser.md#geo-location-filtering).

### Text modifiers

```php
$props->text('category')->keyword();                    // adds .keyword
$props->text('category')->keyword()->makeSortable();    // adds .keyword and .sortable
$props->text('name')->searchAsYouType();                // search_as_you_type mapping
$props->text('suggest')->completion();                  // completion mapping
$props->text('content')->indexPrefixes();               // index_prefixes, 1 to 10 characters
$props->text('content')->indexPrefixes(2, 5);           // index_prefixes, 2 to 5 characters
$props->text('description')->unstructuredText();        // text mapping, the default
```

`keyword()` works only on a `text` mapping. On a `search_as_you_type` or `completion` field it throws an exception. `makeSortable()` returns nothing, so call it last.

A `searchAsYouType()` field runs `multi_match` over the field and its `._2gram` and `._3gram` sub-fields. A `completion()` field feeds [completion suggestions](https://sigmie.com/docs/v2/search.md#suggest-completions).

### Number modifiers

```php
$props->number('count')->integer();
$props->number('rating')->float();
$props->number('views')->long();
$props->number('ratio')->double();
$props->number('amount')->scaledFloat();       // scaling factor 100
$props->number('amount')->scaledFloat(1000);
```

### Range modifiers

```php
$props->range('age_range');                    // integer_range
$props->range('price_range')->float();         // float_range
$props->range('population')->long();           // long_range
$props->range('score')->double();              // double_range
$props->range('availability')->date();         // date_range
$props->range('subnet')->ip();                 // ip_range
```

### Date and datetime

Both methods create an Elasticsearch `date` field that accepts ISO 8601 values. They differ in the check that a collection with properties runs:

| Method | Accepted strings | `DateTime` values become |
|--------|------------------|--------------------------|
| `date()` | `Y-m-d` | `Y-m-d` |
| `datetime()` | `Y-m-d\TH:i:s.uP`, `Y-m-d\TH:i:s.u\Z`, `Y-m-d\TH:i:s.uO`, `Y-m-d\TH:i:s.u`, `Y-m-d\TH:i:sP`, `Y-m-d\TH:i:s\Z`, `Y-m-d\TH:i:sO`, `Y-m-d\TH:i:s` | `Y-m-d\TH:i:s.uP` |

```php
$props->date('release_date');          // '2023-04-07'
$props->datetime('created_at');        // '2023-04-07T12:38:29.000000Z'
```

To add an Elasticsearch date pattern to the mapping, call `format()`. It returns nothing, so call it last:

```php
$props->date('created_at')->format('MM/dd/yyyy');
```

> **Note:** `format()` changes only the mapping. A collection with properties still checks values against the formats in the table above.

## Object and nested types

`object()` and `nested()` take a callback that defines the inner fields. Sigmie searches each inner field with the queries of its own type.

```php
$props->object('director', function (NewProperties $props) {
    $props->name('name');
    $props->number('birth_year')->integer();
    $props->email('contact');
});

$props->nested('cast', function (NewProperties $props) {
    $props->name('actor');
    $props->keyword('character');
    $props->number('screen_time')->integer();
});
```

| Method | Mapping | Use for | Filter syntax |
|--------|---------|---------|---------------|
| `object()` | `object`. Elasticsearch flattens the inner fields. | One object per document. | `director.name:"Nolan"` |
| `nested()` | `nested`. Each item is indexed on its own. | A list of objects whose fields belong together. | `cast:{actor:"Keanu Reeves" AND character:"Neo"}` |

With `object()`, a filter on a list of objects can match `actor` from one item and `character` from another. `nested()` matches both inside one item. The Filter Parser covers [object fields](https://sigmie.com/docs/v2/filter-parser.md#object-fields) and [nested field filtering](https://sigmie.com/docs/v2/filter-parser.md#nested-field-filtering).

## Semantic fields

`semantic()` adds a vector field to a `text` field. Sigmie generates the embeddings when a collection with properties writes the document. It returns the semantic field, so you can chain vector options onto it:

```php
use Sigmie\Enums\VectorSimilarity;

$props->text('description')->semantic(api: 'embeddings', accuracy: 3, dimensions: 384);

$props->text('content')
    ->semantic(api: 'embeddings', dimensions: 512)
    ->similarity(VectorSimilarity::Euclidean);
```

| Parameter | Default | Values |
|-----------|---------|--------|
| `api` | none | The name passed to `$sigmie->registerApi()`. |
| `accuracy` | `3` | `1` to `7`. `1` makes one embedding of the whole value. `2` to `6` embed each chunk and average the vectors, with a denser HNSW graph at each level. `7` keeps every chunk vector and scores them with a script, without a vector index. |
| `dimensions` | `256` | `128`, `256`, `384`, `512`, `1024`, `1536`, `2048`, or `3072`. |

| Option | Default | Values |
|--------|---------|--------|
| `similarity()` | `VectorSimilarity::Cosine` | `Cosine`, `DotProduct`, `Euclidean`, `MaxInnerProduct`. Shortcuts: `cosineSimilarity()`, `dotProductSimilarity()`, `euclideanSimilarity()`, `maxInnerProductSimilarity()`. |
| `indexType()` | `hnsw` | `hnsw`, `int8_hnsw`, `int4_hnsw`, `bbq_hnsw`, `flat`, `int8_flat`, `int4_flat`, `bbq_flat`. |

Register the embeddings API under the name that `api` uses:

```php
use Sigmie\AI\APIs\OpenAIEmbeddingsApi;

$sigmie->registerApi('embeddings', new OpenAIEmbeddingsApi('sk-...'));
```

To add a second vector to the same field, call `semantic()` again on the field:

```php
$description = $props->text('job_description');
$description->semantic(api: 'embeddings', accuracy: 3, dimensions: 512);
$description->semantic(api: 'embeddings', accuracy: 5, dimensions: 512)->euclideanSimilarity();
```

[Semantic Search](https://sigmie.com/docs/v2/semantic-search.md) shows how to search these fields.

## Custom analyzer for one field

`withNewAnalyzer()` gives one `text` field its own analyzer:

```php
use Sigmie\Index\NewAnalyzer;

$props->text('email')
    ->withNewAnalyzer(function (NewAnalyzer $analyzer) {
        $analyzer->tokenizeOnPattern('(@|\.)');
        $analyzer->lowercase();
    });
```

## Custom queries for one field

`withQueries()` replaces the queries that `newSearch()` runs on a field. The callback receives the query string and returns a list of queries:

```php
use Sigmie\Query\Queries\Term\Prefix;
use Sigmie\Query\Queries\Term\Term;
use Sigmie\Query\Queries\Text\Match_;

$props->text('email')
    ->indexPrefixes()
    ->keyword()
    ->withQueries(function (string $queryString) {
        return [
            new Match_('email', $queryString),
            new Prefix('email', $queryString),
            new Term('email.keyword', $queryString),
        ];
    });
```

## Custom property classes

To reuse a field definition, extend `Text` and add it with `type()`. Sigmie calls three methods:

| Method | Called | Purpose |
|--------|--------|---------|
| `configure()` | In the constructor. | Set the mapping and the sub-fields. |
| `analyze()` | When Sigmie builds the analyzers. Only on classes that implement `Analyze`. | Define the field analyzer. |
| `queries()` | For each search with a query string. | Return the queries for the field. |

```php
use Sigmie\Index\NewAnalyzer;
use Sigmie\Mappings\Contracts\Analyze;
use Sigmie\Mappings\Types\Text;
use Sigmie\Query\Queries\Term\Prefix;
use Sigmie\Query\Queries\Term\Term;
use Sigmie\Query\Queries\Text\Match_;

class Color extends Text implements Analyze
{
    public function configure(): void
    {
        $this->unstructuredText()->indexPrefixes()->keyword();
    }

    public function analyze(NewAnalyzer $analyzer): void
    {
        $analyzer->tokenizeOnWhitespaces();
        $analyzer->lowercase();
    }

    public function queries(array|string $queryString): array
    {
        return [
            new Match_($this->name(), $queryString),
            new Prefix($this->name(), $queryString),
            new Term("{$this->name()}.keyword", $queryString),
        ];
    }
}

$props->type(new Color('color'));
```

To ship field types in a package, see [Extending Sigmie](https://sigmie.com/docs/v2/extending.md).

## Inspect properties

`get()` returns the built `Properties`:

```php
$properties = $props->get();

$properties->fieldNames();          // ['title', 'cast.actor', ...]
$properties->fieldNames(true);      // also lists 'cast' and other parent fields
```

Each field checks a value with `validate()`. It returns a pass flag and a message:

```php
[$valid, $message] = $properties['release_date']->validate('release_date', '2023-04-07');
```


---

<!-- source: https://sigmie.com/docs/v2/search -->

# Search

`newSearch()` builds the search a user runs from a search box. Each section on this page solves one task with it. For the boolean query DSL, use [Advanced Queries](https://sigmie.com/docs/v2/query.md) instead.

Every search needs the index properties and a query string:

```php
use Sigmie\Mappings\NewProperties;

$props = new NewProperties;
$props->name();
$props->text('description');

$results = $sigmie->newSearch('fairy-tales')
    ->properties($props)
    ->queryString('snow white')
    ->get();
```

Sigmie reads the properties to decide how to query each field.

## Weight several query strings

Call `queryString()` once per string. The `weight` argument biases the score toward one string:

```php
$sigmie->newSearch('characters')
    ->properties($props)
    ->queryString('Mickey', weight: 2)
    ->queryString('Goofy', weight: 1)
    ->get();
```

## Limit which fields are searched

The search queries every field in the properties. To query only some fields, pass them to `fields()`:

```php
$sigmie->newSearch('fairy-tales')
    ->properties($props)
    ->queryString('Snow White')
    ->fields(['name'])
    ->get();
```

## Limit which fields are returned

To return only some fields of each hit, pass them to `retrieve()`:

```php
$sigmie->newSearch('fairy-tales')
    ->properties($props)
    ->queryString('Snow White')
    ->retrieve(['name', 'description'])
    ->get();
```

To drop fields from every hit, pass them to `except()`. `except()` wins over `retrieve()`, so a nested path stays hidden even when you retrieve its parent:

```php
$sigmie->newSearch('customers')
    ->properties($props)
    ->queryString('jane')
    ->except(['contacts.email'])
    ->get();
```

## Filter

Pass a [filter string](https://sigmie.com/docs/v2/filter-parser.md) to `filters()`. Filters narrow the result set and do not change the score:

```php
$sigmie->newSearch('fairy-tales')
    ->properties($props)
    ->queryString('Sleeping Beauty')
    ->filters('stock>0 AND active:true AND NOT category:"Drama"')
    ->get();
```

### Add a filter the user cannot change

A tenant scope, an ownership check, or an access rule must not come from user input. Pass these as a query object to `filterQuery()`:

```php
use Sigmie\Query\Queries\Term\Term;

$sigmie->newSearch('records')
    ->properties($props)
    ->queryString('checkup')
    ->filters("category:'public' OR category:'private'")   // user input, parsed
    ->filterQuery(new Term('tenant_id', $tenantId))         // never parsed
    ->get();
```

Sigmie combines the `filterQuery()` clause and the parsed filter with AND. The parsed filter stays one clause, so its own `OR` and `NOT` logic does not change. A malformed filter string cannot drop the `filterQuery()` clause, because the parser never reads it. The clause applies to the results, the facet counts, and the semantic (vector) pre-filter.

`filterQuery()` accepts any query clause. Call it once per clause:

```php
use Sigmie\Query\Queries\Term\Terms;

->filterQuery(new Terms('owner_id', $allowedOwnerIds))
```

### Choose strict or lenient parsing

By default, `filters()` is lenient. Sigmie replaces a clause that names an unknown field with a clause that matches nothing, runs the search, and puts the reason under the response's `errors` key:

```php
$response = $sigmie->newSearch('products')
    ->properties($props)
    ->filters('nonexistent:"x"')
    ->get();

$response->json('errors');   // [['message' => 'Field nonexistent does not exist.', ...]]
```

The replaced clause changes the result. After `AND`, it returns no hits. After `NOT`, it removes no hits, so the search returns more than the filter allows. To throw a `ParseException` instead, pass `throwOnError: true`:

```php
->filters('nonexistent:"x"', throwOnError: true)     // throws ParseException
```

[`facets()`](https://sigmie.com/docs/v2/facets.md) takes the same flag for its filter string. Syntax errors, such as a missing operator or an unclosed parenthesis, throw in both modes. See [Filter Parser](https://sigmie.com/docs/v2/filter-parser.md#errors) for the full list.

## Sort

Pass a [sort string](https://sigmie.com/docs/v2/sort-parser.md) to `sort()`:

```php
$sigmie->newSearch('fairy-tales')
    ->properties($props)
    ->queryString('Snow White')
    ->sort('_score:desc name:asc')
    ->get();
```

The default sort is `_score`. Unlike `filters()`, `sort()` throws a `ParseException` on an invalid sort string. With `throwOnError: false`, Sigmie skips the invalid clauses.

## Tolerate typos

List the fields that accept typos in `typoTolerantAttributes()`:

```php
$sigmie->newSearch('fairy-tales')
    ->properties($props)
    ->queryString('Sleping Buety')
    ->typoTolerantAttributes(['name'])
    ->get();
```

Terms with 3 or more characters accept one typo. Terms with 6 or more characters accept two. To change these limits, call `typoTolerance()`:

```php
->typoTolerance(oneTypoChars: 4, twoTypoChars: 8)
```

> **Note:** `typoTolerance()` only sets the limits. Fields that are not in `typoTolerantAttributes()` always match exactly.

## Highlight matches

Pass the fields and the HTML that wraps each match to `highlighting()`:

```php
$response = $sigmie->newSearch('fairy-tales')
    ->properties($props)
    ->queryString('sleeping beauty')
    ->highlighting(
        ['name'],
        prefix: '<mark>',
        suffix: '</mark>',
    )
    ->get();

$response->json('hits.0.highlight.name');   // fragments with each match in <mark>
```

`prefix` and `suffix` have no default. Each highlighted field returns up to three fragments of 150 characters.

## Weight fields

Pass a weight per field to `weight()`:

```php
$sigmie->newSearch('fairy-tales')
    ->properties($props)
    ->queryString('sleeping beauty')
    ->weight(['name' => 4, 'description' => 1])
    ->get();
```

A match in `name` now scores 4 times higher than the same match in `description`.

## Drop low-scoring hits

Pass the lowest score to keep to `minScore()`:

```php
$sigmie->newSearch('fairy-tales')
    ->properties($props)
    ->queryString('Mickey')
    ->weight(['name' => 5])
    ->minScore(2)
    ->get();
```

## Paginate

`size()` sets the hits per page and defaults to 20. `from()` sets how many hits to skip. This search returns the second page of 10:

```php
$sigmie->newSearch('fairy-tales')
    ->properties($props)
    ->queryString('sleeping beauty')
    ->from(10)
    ->size(10)
    ->get();
```

`page()` takes a page number and a page size instead:

```php
->page(2, 20)               // same as from(20)->size(20)
```

## Return one hit per group

To return one hit per value of a field, such as one hit per product across its variants, pass the field to `uniqueBy()`:

```php
$sigmie->newSearch('products')
    ->properties($props)
    ->queryString('sneakers')
    ->uniqueBy('product_id')
    ->get();
```

To also return the next best hits of each group as inner hits, pass `top`:

```php
->uniqueBy('product_id', top: 3)
```

The field must hold one value per document, such as a `keyword`.

## Add facets

Pass the facet fields to `facets()`. See [Facets](https://sigmie.com/docs/v2/facets.md) for the syntax:

```php
$response = $sigmie->newSearch('products')
    ->properties($props)
    ->queryString('laptop')
    ->facets('brand category price:100')
    ->get();

$facets = $response->json('facets');
```

## Add semantic search

Call `semantic()` to match by vector as well as by keyword:

```php
$sigmie->newSearch('articles')
    ->properties($props)
    ->semantic()
    ->queryString('artificial intelligence')
    ->get();
```

To match by vector only, also call `disableKeywordSearch()`:

```php
->semantic()->disableKeywordSearch()
```

See [Semantic Search](https://sigmie.com/docs/v2/semantic-search.md) to set up embeddings.

## Suggest completions

Autocomplete reads the first `completion` field in the properties. Pass the typed prefix to `autocompletePrefix()`:

```php
$props = new NewProperties;
$props->completion('title');

$response = $sigmie->newSearch('movies')
    ->properties($props)
    ->autocompletePrefix('star')
    ->autocompleteSize(3)
    ->queryString('')
    ->get();

$response->json('autocomplete.0.options');   // [['text' => 'Star Trek', ...], ...]
```

`autocompleteSize()` defaults to 5.

## Search several indices

Pass the index names to `newSearch()`, separated by commas:

```php
$result = $sigmie->newSearch("$germanIndex,$englishIndex")
    ->properties($props)
    ->queryString('door tür')
    ->get();
```

## Search nested fields

Name a field inside a nested or object field with dot notation:

```php
$sigmie->newSearch('users')
    ->properties($props)
    ->queryString('Pluto')
    ->fields(['contact.dog.name'])
    ->retrieve(['contact.dog.name'])
    ->get();
```

Without `fields()`, the search also queries every field inside nested fields.

## Matching nested items (inner hits)

A nested field holds a list of objects, such as the reviews of a product. When a search for "battery" matches one review, Elasticsearch returns the whole product. Its `_source` holds every review, so you cannot see which review matched. Inner hits return, per hit, the nested items that matched:

```
product "Aurora Phone"           search "battery"
  reviews[0] Battery easily...   -> matched
  reviews[1] Camera is great...  -> not matched
  reviews[2] Battery drains...   -> matched

_source          -> the product, all 3 reviews
_matches.reviews -> reviews 0 and 2, total 2
```

The examples below use a shop index with nested reviews, questions, and answers:

```php
$props = new NewProperties;
$props->title('title');
$props->nested('reviews', function (NewProperties $props) {
    $props->text('comment');
    $props->number('stars');
    $props->keyword('author');
});
$props->nested('questions', function (NewProperties $props) {
    $props->text('text');
});
$props->nested('answers', function (NewProperties $props) {
    $props->text('text');
});

$sigmie->newIndex('products')->properties($props)->lowercase()->create();

$sigmie->collect('products', refresh: true)->merge([
    new Document([
        'title' => 'Aurora Phone',
        'reviews' => [
            ['comment' => 'Battery easily lasts two days.', 'stars' => 5, 'author' => 'anna'],
            ['comment' => 'Camera is great in daylight.', 'stars' => 4, 'author' => 'ben'],
            ['comment' => 'Battery drains fast when gaming.', 'stars' => 2, 'author' => 'chris'],
        ],
        'questions' => [
            ['text' => 'Is the battery replaceable?'],
            ['text' => 'Does it support eSIM?'],
        ],
        'answers' => [
            ['text' => 'No, the battery is sealed.'],
            ['text' => 'Yes, eSIM works.'],
        ],
    ], 'aurora'),
]);
```

Pass the nested path to `innerHits()`:

```php
$hits = $sigmie->newSearch('products')
    ->properties($props)
    ->queryString('battery')
    ->innerHits('reviews')
    ->get()
    ->json('hits');
```

Each hit keeps its `_source` and gets a `_matches` key:

```json
{
    "_id": "aurora",
    "_source": {
        "title": "Aurora Phone",
        "reviews": [
            {"comment": "Battery easily lasts two days.", "stars": 5, "author": "anna"},
            {"comment": "Camera is great in daylight.", "stars": 4, "author": "ben"},
            {"comment": "Battery drains fast when gaming.", "stars": 2, "author": "chris"}
        ],
        "questions": [
            {"text": "Is the battery replaceable?"},
            {"text": "Does it support eSIM?"}
        ],
        "answers": [
            {"text": "No, the battery is sealed."},
            {"text": "Yes, eSIM works."}
        ]
    },
    "_matches": {
        "reviews": {
            "total": 2,
            "items": [
                {"_offset": 0, "comment": "Battery easily lasts two days.", "stars": 5, "author": "anna"},
                {"_offset": 2, "comment": "Battery drains fast when gaming.", "stars": 2, "author": "chris"}
            ]
        }
    }
}
```

| Key | Holds |
|-----|-------|
| `_source` | The document, or the `retrieve()` fields. Inner hits never change it. |
| `_matches.reviews.total` | How many reviews of this product matched. |
| `_matches.reviews.items` | The matching reviews, best match first. |
| `_offset` | The item's position in the `reviews` array, starting at 0. |

Only the paths you name get matches. Without `innerHits()`, a hit has no `_matches` key. A requested path with no matching items still appears, with `total: 0` and no items.

### Read matches from a hit

Each `Hit` from `hits()` reads its matches by nested path:

```php
$response = $sigmie->newSearch('products')
    ->properties($props)
    ->queryString('battery')
    ->innerHits('reviews.comment')
    ->innerHits('questions.text')
    ->innerHits('answers')
    ->get();

foreach ($response->hits() as $hit) {
    $hit->matches('reviews');      // [['_offset' => 0, 'comment' => 'Battery easily lasts two days.'],
                                   //  ['_offset' => 2, 'comment' => 'Battery drains fast when gaming.']]
    $hit->matchesTotal('reviews'); // 2
    $hit->matches();               // ['reviews' => [...], 'questions' => [...], 'answers' => [...]]
}
```

| Method | Returns |
|--------|---------|
| `matches(string $path)` | The matching items of that nested path. |
| `matchesTotal(string $path)` | How many items of that path matched, including items beyond `size`. |
| `matches()` | The items of every requested path, keyed by path. |

The argument is the nested path, the same key as in `_matches`. A field inside the path resolves to the path, so `matches('questions.text')` returns the `questions` items. A path you did not request returns `[]`, and its total is `0`. `Hit::toArray()` includes `_matches` only when the search requested inner hits. Reranked hits keep their matches.

The raw form is the `_matches` key of `json('hits')`:

```php
$response->json('hits.0._matches.reviews')['total']; // 2
```

### Return only some fields of each item

Name a field inside the nested field to return only that field of each matching item. Call `innerHits()` again for another field of the same path, and the fields merge:

```php
->innerHits('reviews.comment')
->innerHits('reviews.stars')
```

```json
"_matches": {
    "reviews": {
        "total": 2,
        "items": [
            {"_offset": 0, "comment": "Battery easily lasts two days.", "stars": 5},
            {"_offset": 2, "comment": "Battery drains fast when gaming.", "stars": 2}
        ]
    }
}
```

### Return only the matching items

`retrieve()` controls `_source`, and `innerHits()` controls `_matches`. To get the product title and only the matching reviews, leave the nested field out of `retrieve()`:

```php
$hits = $sigmie->newSearch('products')
    ->properties($props)
    ->queryString('battery')
    ->retrieve(['title'])
    ->innerHits('reviews.comment')
    ->get()
    ->json('hits');
```

```json
{
    "_id": "aurora",
    "_source": {"title": "Aurora Phone"},
    "_matches": {
        "reviews": {
            "total": 2,
            "items": [
                {"_offset": 0, "comment": "Battery easily lasts two days."},
                {"_offset": 2, "comment": "Battery drains fast when gaming."}
            ]
        }
    }
}
```

### Request several paths

Call `innerHits()` once per nested path. Each path gets its own entry and its own `size`:

```php
$hits = $sigmie->newSearch('products')
    ->properties($props)
    ->queryString('battery')
    ->retrieve(['title'])
    ->innerHits('reviews', size: 10)
    ->innerHits('questions.text', size: 5)
    ->innerHits('answers')                 // default size: 100
    ->get()
    ->json('hits');
```

```json
"_matches": {
    "reviews": {
        "total": 2,
        "items": [
            {"_offset": 0, "comment": "Battery easily lasts two days.", "stars": 5, "author": "anna"},
            {"_offset": 2, "comment": "Battery drains fast when gaming.", "stars": 2, "author": "chris"}
        ]
    },
    "questions": {
        "total": 1,
        "items": [
            {"_offset": 0, "text": "Is the battery replaceable?"}
        ]
    },
    "answers": {
        "total": 1,
        "items": [
            {"_offset": 0, "text": "No, the battery is sealed."}
        ]
    }
}
```

To give several fields or paths one size, pass an array:

```php
->innerHits(['reviews.comment', 'reviews.stars', 'questions.text'], size: 5)
```

### Cap the number of items

`innerHits()` returns up to 100 matching items per path. 100 is the Elasticsearch default for `index.max_inner_result_window`. To return fewer, pass `size`:

```php
->retrieve(['title'])
->innerHits('reviews.comment', size: 1)
```

`total` still counts every match. Here `total` is 2 and one item returns:

```json
"_matches": {
    "reviews": {
        "total": 2,
        "items": [
            {"_offset": 0, "comment": "Battery easily lasts two days."}
        ]
    }
}
```

When `total` is greater than the number of items, the list is capped. With more than 100 matches, `items` holds 100 and `total` holds the real count. To return more than 100, raise `index.max_inner_result_window` on the index and pass a larger `size`.

When you call `innerHits()` again for the same path, the later `size` wins and the fields merge. A whole-path request covers every field under it, so `->innerHits('reviews.comment', size: 8)->innerHits('reviews', size: 1)` returns one whole review.

### Combine with nested filters

A [nested filter](https://sigmie.com/docs/v2/filter-parser.md#nested-field-filtering) also produces matches. When the query and a filter both match items of one path, `_matches` holds each item once:

```php
->queryString('battery')                // matches reviews 0 and 2
->filters('reviews:{stars>=4}')          // matches reviews 0 and 1
->innerHits('reviews.comment')           // reviews 0, 1 and 2, total 3
```

Deeper nested paths use dots too. For orders with nested items, `innerHits('orders.items.sku')` with the filter `orders:{items:{sku:'y'}}` returns the matching items under `_matches['orders.items']`. Each item also holds `_parents`, the offsets of its parent items, such as `{"orders": 1}`.

`except()` applies to the items too. With `->except(['reviews.author'])`, no item holds `author`, even with `innerHits('reviews')`.

> **Note:** `innerHits()` throws an `InvalidArgumentException` for a field that does not exist or is not inside a nested field, such as `title`.

## Read the response

`get()` returns the response:

```php
$response = $sigmie->newSearch('fairy-tales')
    ->properties($props)
    ->queryString('mickey')
    ->get();

$response->total();                  // total matching documents
$response->hits();                   // array of Hit objects
$response->json('hits');             // raw hits array
$response->json('hits.0._source');   // one value, by dot notation
```

## Return nothing for an empty query

An empty query string returns every document. To return no documents, call `noResultsOnEmptySearch()`:

```php
->noResultsOnEmptySearch()
```

## Run the search asynchronously

`promise()` returns a Guzzle promise and does not wait for the response:

```php
$promise = $sigmie->newSearch('fairy-tales')
    ->properties($props)
    ->queryString('mickey')
    ->promise();
```

## Iterating over all matching hits

`size()` sets one page. To read every matching document, for an export or a migration, call `each()` or `lazy()`. Both keep your query string, filters, and fields. They page with a point in time (PIT) and `search_after`, so writes during the iteration do not break it.

`each()` calls a function for each hit:

```php
use Sigmie\Document\Hit;

$sigmie->newSearch('orders')
    ->properties($props)
    ->filters('status:completed')
    ->each(function (Hit $hit) use ($csv): void {
        $csv->writeRow($hit->_source);
    });
```

`lazy()` returns a generator:

```php
$generator = $sigmie->newSearch('orders')
    ->properties($props)
    ->filters('status:completed')
    ->lazy();

foreach ($generator as $hit) {
    processHit($hit);
}
```

Each `Hit` has `_id`, `_source`, and `_score`.

> **Note:** `each()` and `lazy()` ignore `from()`, `size()`, `page()`, and `highlighting()`.

### Set the page size

Each request fetches 500 hits by default. To change it, call `chunk()`. A smaller chunk uses less memory and sends more requests:

```php
$sigmie->newSearch('products')
    ->properties($props)
    ->chunk(100)
    ->each(function (Hit $hit): void {
        // 100 at a time
    });
```

### Sort the iteration

A point in time needs a sort with a unique last key. Sigmie adds one, `_shard_doc` on Elasticsearch and `_id` on OpenSearch, unless your sort already ends with it.

| Builder | Where the sort comes from |
|---------|---------------------------|
| `newSearch()` | The `sort()` string. A sort on `_score` or `_doc` alone becomes the tiebreaker only. |
| `newQuery()` | `sortString()` or `sort()`. Without a sort, hits arrive in a stable but unranked order. |
| `raw()` | The top-level `sort` key of the body. |

For `newQuery()`, sort on fields that exist in the mapping:

```php
$multi->raw('orders', [
    'query' => ['match_all' => (object) []],
    'sort' => [['processed_at' => 'asc']],
]);
```

With `collapse` in the body, Sigmie adds no tiebreaker. Elasticsearch allows one sort key with `collapse` and `search_after`, so you choose that key.

### Iterate over a multi-search

`newMultiSearch()` sends several searches in one `_msearch` request, and `get()` returns one page of each. To read every hit of every search, call `each()` or `lazy()` on the multi-search:

```php
use Sigmie\Document\Hit;

$multi = $sigmie->newMultiSearch();

$multi->newSearch('orders')
    ->properties($orderProps)
    ->filters('status:pending')
    ->chunk(200);

$multi->newQuery('products')->matchAll();

$multi->raw('orders', [
    'query' => ['term' => ['status' => 'pending']],
])->chunk(200);

foreach ($multi->lazy() as $hit) {
    exportRow($hit);
}
```

Each search runs its own point in time, and hits arrive in the order you added the searches. Call `chunk()` on each search. The multi-search has no shared chunk size.


---

<!-- source: https://sigmie.com/docs/v2/query -->

# Advanced Queries

`newQuery()` builds an Elasticsearch query clause by clause. Use it for queries that [`newSearch()`](https://sigmie.com/docs/v2/search.md) does not build, such as custom boolean logic and script scoring. `newSearch()` has typo tolerance, highlighting, facets, and semantic search. `newQuery()` has none of these.

```php
use Sigmie\Query\Queries\Compound\Boolean;

$response = $sigmie->newQuery('disney-movies')
    ->properties($props)
    ->sortString('name:asc')
    ->bool(function (Boolean $bool) {
        $bool->filter()->term('is_active', true);
        $bool->must()->multiMatch(['name', 'description'], 'goofy');
        $bool->mustNot()->wildcard('name', 'test*');
        $bool->should()->bool(fn (Boolean $bool) =>
            $bool->must()->match('name', 'Mickey')
        );
    })
    ->from(0)
    ->size(15)
    ->get();
```

## Builder

`$sigmie->newQuery($index)` returns a `NewQuery`. Call the setup methods first, then one query method. The query method returns a `Search`, which takes the request options and runs the request.

```
newQuery('movies')            NewQuery
  ->properties($props)          setup
  ->sortString('year:desc')     setup
  ->bool(...)                 query method, returns Search
  ->size(20)                    request option
  ->get()                     runs the request
```

| `NewQuery` method | Does |
|-------------------|------|
| `properties($props)` | Sets the properties that `parse()`, `sortString()`, and `postFilterString()` read. |
| `sortString(string $sort)` | Sets the sort from a [sort string](https://sigmie.com/docs/v2/sort-parser.md). |
| `sort(array $sort)` | Sets the sort from an Elasticsearch `sort` array. |
| `chunk(int $size = 500)` | Sets the page size for `each()` and `lazy()`. |
| `each(Closure $fn)`, `lazy()` | Read every matching hit. See [Iterating over all matching hits](https://sigmie.com/docs/v2/search.md#iterating-over-all-matching-hits). |

The query methods, such as `term()` and `match()`, take Elasticsearch field names as they are. Only `parse()`, `sortString()`, and `postFilterString()` read the properties.

## Boolean clauses

`bool()` passes a `Boolean` to your function. Each of its four methods adds clauses of one kind:

| Clause | Matches when | Affects `_score` |
|--------|--------------|------------------|
| `must()` | Every clause matches. | Yes |
| `filter()` | Every clause matches. | No |
| `mustNot()` | No clause matches. | No |
| `should()` | See below. | Yes |

```php
$sigmie->newQuery('movies')
    ->properties($props)
    ->bool(function (Boolean $bool) {
        $bool->must()->match('title', 'matrix');
        $bool->filter()->range('year', ['>' => 1990]);
        $bool->should()->term('genre', 'sci-fi');
        $bool->mustNot()->term('rating', 'R');
    })
    ->get();
```

`should()` clauses follow the Elasticsearch rule. Without `must()` or `filter()` clauses, at least one `should()` clause must match. With them, `should()` clauses only raise the score. To require a number of `should()` matches, set `minimum_should_match`:

```php
$bool->addRaw('minimum_should_match', 1);
```

Use `filter()` instead of `must()` when a clause must not change the score. Elasticsearch can cache filter clauses.

A clause builder accepts the query methods below, plus these:

| Method | Adds |
|--------|------|
| `bool(callable $fn, float $boost = 1)` | A nested `bool` query. |
| `query(QueryClause $query)` | Any query object. |
| `parse(string $filter)` | The query from a [filter string](https://sigmie.com/docs/v2/filter-parser.md). Returns nothing, so do not chain after it. |

## Query methods

Each method is available on `NewQuery` and on the clause builders. On `NewQuery`, the method sets the whole query and returns a `Search`. Every method also takes an optional `float $boost = 1`, except `term()` on `NewQuery`.

| Method | Elasticsearch query |
|--------|---------------------|
| `matchAll()` | `match_all` |
| `matchNone()` | `match_none` |
| `term(string $field, $value)` | `term` |
| `terms(string $field, array $values)` | `terms` |
| `match(string $field, string $query)` | `match` |
| `multiMatch(array $fields, string $query)` | `multi_match` |
| `range(string $field, array $values)` | `range` |
| `exists(string $field)` | `exists` |
| `ids(array $ids)` | `ids` |
| `regex(string $field, string $regex)` | `regexp` |
| `wildcard(string $field, string $value)` | `wildcard` |
| `fuzzy(string $field, string $value)` | `fuzzy` |

`NewQuery` also has `bool()`, `query()`, and `parse()`, with the same meaning as on the clause builders. `postFilter(QueryClause $query)` and `postFilterString(string $filter)` set a post filter.

### Term and terms

`term()` matches one exact value. Use it on `keyword`, `bool`, and number fields:

```php
$query->term('active', true);
$query->term('user_id', 13);
```

`terms()` matches any of several values:

```php
$query->terms('category', ['horror', 'action']);
```

> **Note:** Elasticsearch splits a `text` field into tokens, so `term()` on it rarely matches. Add a `keyword()` sub-field and query that:
>
> ```php
> $props->text('category')->keyword();
> // then:
> $query->term('category.keyword', 'drama');
> ```

### Match and multi-match

`match()` analyzes the query text first. Use it on `text` fields:

```php
$query->match('name', 'mickey');
$query->multiMatch(['name', 'username'], 'mickey');
```

### Range

```php
$query->range('count', ['>=' => 233]);
$query->range('price', ['>=' => 30, '<=' => 130]);
```

The operators are `>=`, `>`, `<=`, and `<`.

### Exists and IDs

`exists()` matches documents with any value in the field. `ids()` matches documents by `_id`:

```php
$query->exists('director');
$query->ids(['dkKwMe4UBAUb2dMteRe2', 'wd6Me4UBAUb2dMJT']);
```

### Regex, wildcard, and fuzzy

```php
$query->regex('category', '(horror|action)');
$query->wildcard('name', 'john*');
$query->fuzzy('name', 'john');
```

### Boost

`boost` scales a clause's share of `_score`:

```php
$query->matchAll(boost: 5);
```

## Filter strings

`parse()` builds the query from a [filter string](https://sigmie.com/docs/v2/filter-parser.md). It reads the properties and throws a `ParseException` on any error:

```php
$sigmie->newQuery('movies')
    ->properties($props)
    ->parse('name:"John Doe" AND age<21')
    ->get();
```

## Script scoring

Wrap a query in a `FunctionScore` to score it with a Painless script, then pass it to `query()`:

```php
use Sigmie\Query\FunctionScore;
use Sigmie\Query\Queries\MatchAll;

$sigmie->newQuery('movies')
    ->query(new FunctionScore(
        new MatchAll,
        source: "Math.log(2 + doc['popularity'].value)",
        boostMode: 'replace',
    ))
    ->get();
```

`boostMode` defaults to `multiply`. `params` passes values to the script.

## Aggregations

`aggregate()` on the returned `Search` adds aggregations:

```php
use Sigmie\Query\Aggs;

$response = $sigmie->newQuery('orders')
    ->matchAll()
    ->aggregate(function (Aggs $aggs) {
        $aggs->sum(name: 'turnover', field: 'price');
    })
    ->get();

$response->aggregation('turnover.value');
```

See [Aggregations](https://sigmie.com/docs/v2/aggregations.md) for every aggregation type. `newQuery()` has no `facets()` method. For facets, use [`newSearch()`](https://sigmie.com/docs/v2/search.md#add-facets).

## Sorting

`sortString()` takes a [sort string](https://sigmie.com/docs/v2/sort-parser.md). `sort()` takes an Elasticsearch `sort` array:

```php
$query->sortString('name:asc created_at:desc');
$query->sort([['year' => 'desc'], ['_score' => 'desc']]);
```

Both are on `NewQuery` and on the returned `Search`. Each call replaces the previous sort, so put every field in one call. `sortString()` throws a `ParseException` on an invalid sort, such as `_score:asc`.

> **Note:** A `text` field sorts only with a `.sortable` sub-field. Add it with `$props->text('name')->makeSortable()`.

## Request options

These methods are on the `Search` that the query method returns:

| Method | Does | Default |
|--------|------|---------|
| `from(int $from)` | Skips hits. | `0` |
| `size(int $size)` | Sets the number of hits. | `500` |
| `minScore(float $score)` | Drops hits below the score. | `0` |
| `fields(array $fields)` | Returns only these `_source` fields. | All |
| `except(array $fields)` | Drops these `_source` fields. | None |
| `highlight(array $highlight)` | Sets the Elasticsearch `highlight` body. | None |
| `trackTotalHits(bool\|int $trackTotalHits = true)` | Sets `track_total_hits`. | Not sent |
| `addRaw(string $key, mixed $value)` | Adds a top-level key to the request body. | None |

```php
$response = $sigmie->newQuery('movies')
    ->properties($props)
    ->sortString('title:asc')
    ->matchAll()
    ->from(0)
    ->size(20)
    ->get();
```

## Responses

`get()` runs the request and returns the response. `promise()` returns a Guzzle promise instead.

```php
$response->json();                       // full response
$response->json('hits.hits');            // hits array
$response->total();                      // hits.total.value
$response->hits();                       // array of Hit objects
$response->aggregation('turnover.value');
```

## Debugging

`getDSL()` on the `Search` returns the request body as an array:

```php
$dsl = $sigmie->newQuery('movies')->matchAll()->getDSL();
```


---

<!-- source: https://sigmie.com/docs/v2/semantic-search -->

# Semantic Search

Semantic search matches documents by meaning. The query "portable computer for work" finds a document about a "laptop" even though the two share no word.

Sigmie embeds the text of each semantic field when you index a document, embeds the query when you search, and returns the documents whose vectors are closest to the query vector. This page shows how to set that up and tune it.

## Register an embeddings API

Register a client for your embeddings provider under a name of your choice:

```php
use Sigmie\AI\APIs\OpenAIEmbeddingsApi;

$sigmie->registerApi('embeddings', new OpenAIEmbeddingsApi('sk-...'));
```

Your field definitions refer to the API by this name. Sigmie ships these clients, and all of them implement `Sigmie\AI\Contracts\EmbeddingsApi`:

```php
use Sigmie\AI\APIs\CohereEmbeddingsApi;
use Sigmie\AI\APIs\InfinityEmbeddingsApi;
use Sigmie\AI\APIs\OpenAIEmbeddingsApi;
use Sigmie\AI\APIs\VoyageEmbeddingsApi;
use Sigmie\Enums\CohereInputType;

$sigmie->registerApi('embeddings', new OpenAIEmbeddingsApi('sk-...'));
$sigmie->registerApi('embeddings', new CohereEmbeddingsApi('co-...', CohereInputType::SearchDocument));
$sigmie->registerApi('embeddings', new VoyageEmbeddingsApi('pa-...'));

// Local Infinity service (see Docker docs)
$sigmie->registerApi('embeddings', new InfinityEmbeddingsApi(
    baseUrl: 'http://localhost:7997',
    model: 'BAAI/bge-small-en-v1.5',
));
```

To run a local Infinity service, see [Docker](https://sigmie.com/docs/v2/docker.md).

### Use another provider

To use a provider that Sigmie has no client for, implement `EmbeddingsApi` and register your class:

```php
use GuzzleHttp\Promise\Promise;
use Sigmie\AI\Contracts\EmbeddingsApi;

class MyEmbeddings implements EmbeddingsApi
{
    public function embed(string $text, int $dimensions): array { /* ... */ }
    public function batchEmbed(array $payload): array { /* ... */ }
    public function promiseEmbed(string $text, int $dimensions): Promise { /* ... */ }
    public function model(): string { /* ... */ }
    public function maxBatchSize(): int { /* ... */ }
}

$sigmie->registerApi('embeddings', new MyEmbeddings());
```

`maxBatchSize()` returns the number of texts your provider accepts in one `batchEmbed()` call.

## Define semantic fields

Call `semantic()` on a text field. Set `api` to the registered name and `dimensions` to the vector size of your model:

```php
use Sigmie\Mappings\NewProperties;

$props = new NewProperties;
$props->title('title')->semantic(api: 'embeddings', dimensions: 1536);
$props->text('description')->semantic(api: 'embeddings', dimensions: 1536);

$sigmie->newIndex('articles')->properties($props)->create();
```

OpenAI's `text-embedding-3-small` returns 1536 dimensions. Infinity's `bge-small-en-v1.5` returns 384. `dimensions` defaults to 256. `semantic()` accepts 128, 256, 384, 512, 1024, 1536, 2048, or 3072 and throws for any other value. For another size, set it with `dimensions()` in [`newSemantic()`](#fine-tune-the-vector-field).

## Index documents

Add documents with `merge()` or `add()`. Sigmie calls the embeddings API for every semantic field before it writes the documents:

```php
use Sigmie\Document\Document;

$sigmie->collect('articles', refresh: true)
    ->properties($props)
    ->merge([
        new Document([
            'title' => 'Introduction to Machine Learning',
            'description' => 'A primer on supervised and unsupervised learning.',
        ]),
        new Document([
            'title' => 'Deep Learning Fundamentals',
            'description' => 'Neural networks form the basis of deep learning.',
        ]),
    ]);
```

`merge()` sends the texts of many documents in batched requests, not one request per document. Sigmie groups texts that share the same API, dimensions, and input type (text or image), then splits each group into chunks of `min(100, maxBatchSize())` texts:

```php
$articles->merge($thirtyDocs);     // 1 request of 30
$articles->merge($twoHundredDocs); // 2 requests: 100 + 100
```

The provider limits are 2048 for OpenAI and Jina, 512 for Infinity, 128 for Voyage, and 96 for Cohere.

> **Warning:** If one chunk fails, the whole `merge()` fails. Sigmie does not retry or write part of the batch. Keep each `merge()` call to a size your provider serves reliably.

## Search by meaning

Call `semantic()` on the search:

```php
$response = $sigmie->newSearch('articles')
    ->properties($props)
    ->semantic()
    ->queryString('artificial intelligence basics')
    ->get();
```

The search runs the keyword query and the vector query together. A document that matches both ranks above a document that matches only one.

To rank by vectors only, turn off the keyword query:

```php
$sigmie->newSearch('articles')
    ->properties($props)
    ->semantic()
    ->disableKeywordSearch()
    ->queryString('machine learning algorithms')
    ->get();
```

To give one side more weight, multiply its score. Both multipliers default to `1.0`:

```php
$sigmie->newSearch('articles')
    ->properties($props)
    ->semantic()
    ->textScoreMultiplier(1.0)
    ->semanticScoreMultiplier(2.0)
    ->queryString('quantum computing')
    ->get();
```

To search some semantic fields and skip the others, list them with `fields()`:

```php
$sigmie->newSearch('articles')
    ->properties($props)
    ->semantic()
    ->fields(['title'])
    ->queryString('deep learning neural networks')
    ->get();
```

Filters, sorting, and facets work with semantic search the same way they work with keyword search. See [Search](https://sigmie.com/docs/v2/search.md). A multilingual embedding model also matches across languages, so the query "machine learning" finds "apprentissage automatique".

## Choose an accuracy level

`accuracy` goes from 1 to 7 and defaults to 3. It sets how Sigmie embeds a field and the HNSW graph parameters `m` and `ef_construction`. A higher level gives better recall and costs more memory and indexing time:

```php
$props->text('content')->semantic(api: 'embeddings', dimensions: 512, accuracy: 5);
```

| Accuracy | Embedding | `m` at 256 dims | `ef_construction` at 256 dims |
|----------|-----------|-----------------|-------------------------------|
| 1 | One vector for all values joined | 12 | 60 |
| 2 | One vector per value, averaged | 16 | 100 |
| 3 (default) | One vector per value, averaged | 24 | 150 |
| 4 | One vector per value, averaged | 32 | 200 |
| 5 | One vector per value, averaged | 40 | 300 |
| 6 | One vector per value, averaged | 48 | 400 |
| 7 | One vector per value, exact script score, no HNSW graph | none | none |

For other dimensions, Sigmie multiplies both values by `sqrt(dimensions / 256)`, then caps `m` at 64 and `ef_construction` at 500. Accuracy 3 at 1536 dimensions gives `m` 59 and `ef_construction` 367.

Use a high level on the fields that carry the meaning, such as titles and main content. Use a low level on long supporting text.

## Search array values

A semantic field can hold an array. At accuracy 2 and above, Sigmie embeds each value on its own. At accuracy 1, it joins the values into one string and embeds that:

```php
new Document([
    'experience' => [
        'Artist',
        'Graphic Design',
        'Creative Director',
    ],
]);

// "drawing illustration" matches "Artist"
$sigmie->newSearch('professionals')
    ->properties($props)
    ->semantic()
    ->queryString('drawing illustration')
    ->get();
```

## Fine-tune the vector field

For settings that `semantic()` does not take, call `newSemantic()` and configure a `NewSemanticField`:

```php
use Sigmie\Enums\VectorSimilarity;
use Sigmie\Mappings\NewSemanticField;

$props->text('content')->newSemantic(function (NewSemanticField $semantic): void {
    $semantic->api('embeddings')
        ->accuracy(3, 1024)
        ->similarity(VectorSimilarity::DotProduct);
});
```

`semantic()` returns the same `NewSemanticField`, so you can also chain onto it: `->semantic(api: 'embeddings', dimensions: 512)->euclideanSimilarity()`.

### Similarity

Similarity defaults to cosine. Set it with `similarity()` or a shortcut method:

| `VectorSimilarity` case | Shortcut | Elasticsearch value |
|-------------------------|----------|---------------------|
| `Cosine` | `cosineSimilarity()` | `cosine` |
| `DotProduct` | `dotProductSimilarity()` | `dot_product` |
| `Euclidean` | `euclideanSimilarity()` | `l2_norm` |
| `MaxInnerProduct` | `maxInnerProductSimilarity()` | `max_inner_product` |

Cosine fits most text models. Dot product needs normalized vectors, and Sigmie normalizes vectors by default.

### Index type

Sigmie indexes vectors as float `hnsw` by default. On Elasticsearch, a quantized index type keeps a compressed copy of each vector in memory:

```php
$props->text('content')->newSemantic(function (NewSemanticField $semantic): void {
    $semantic->api('embeddings')->dimensions(768)->indexType('bbq_hnsw');
});
```

The allowed types are `hnsw`, `int8_hnsw`, `int4_hnsw`, `bbq_hnsw`, `flat`, `int8_flat`, `int4_flat`, and `bbq_flat`. Flat types skip the HNSW graph, so Sigmie does not send `m` and `ef_construction`.

Each index type gets its own vector field name, for example `bbq_hnsw_m64_efc300_dims768_cosine_concat`. A new type adds a new field, so reindex your documents after you change it. OpenSearch supports only `hnsw`. See [OpenSearch](https://sigmie.com/docs/v2/opensearch.md).

### Several vectors on one field

Call `semantic()` or `newSemantic()` more than once on the same field. Each call adds one vector:

```php
$jobDescription = $props->text('job_description');

$jobDescription->semantic(api: 'embeddings', dimensions: 512, accuracy: 3);
$jobDescription->semantic(api: 'embeddings', dimensions: 512, accuracy: 5)
    ->euclideanSimilarity();
```

## Supply your own embeddings

To skip the embeddings API, for example in a backfill, put the vectors in the document under `_embeddings`. Key them by field path, then by vector field name:

```php
new Document([
    'title' => 'AI Research Paper',
    '_embeddings' => [
        'title' => [
            'm24_efc150_dims256_cosine_avg' => [0.1, 0.2, 0.3, /* ... */],
        ],
    ],
]);
```

Sigmie skips a field only when the document has a vector for every vector field of that field. It embeds the other fields as usual. To list the vector field names of a field:

```php
$props->get()->get('title')->vectorFields()
    ->map(fn ($vector) => $vector->name)
    ->toArray();
// ['m24_efc150_dims256_cosine_avg']
```

## Rerank the top results

A cross-encoder reranker scores each hit against the query and gives a better order for the top results. Register a rerank API, fetch more hits than you need, then call `rerank()` on the response:

```php
use Sigmie\AI\APIs\InfinityRerankApi;

$sigmie->registerApi('my-rerank', new InfinityRerankApi(
    baseUrl: 'http://localhost:7998',
    model: 'cross-encoder/ms-marco-MiniLM-L-6-v2',
));

$response = $sigmie->newSearch('articles')
    ->properties($props)
    ->semantic()
    ->queryString('quantum computing applications')
    ->size(20)
    ->get();

$top5 = $response->rerank('my-rerank', ['content'], topK: 5);
```

[Retrieval and Agents](https://sigmie.com/docs/v2/rag.md) covers rerankers and passing results to a language model.

To find documents similar to documents you already have, use [Recommendations](https://sigmie.com/docs/v2/recommendations.md).

## Troubleshoot

**No results.** Fetch one document with `$sigmie->collect('articles')->get($id)` and check that it has an `_embeddings` key. If it does not, check that the `api` name on the field matches the name you registered.

**Slow searches.** Lower the accuracy or the dimensions. Add filters so that fewer documents reach vector ranking.

**High memory use.** Lower the accuracy or the dimensions, or use a quantized index type.


---

<!-- source: https://sigmie.com/docs/v2/aggregations -->

# Aggregations

`aggregate()` on a query adds Elasticsearch aggregations to the request. Each method on the `Aggs` builder adds one aggregation and returns it, so you can configure it further.

For filter sidebars, use [Facets](https://sigmie.com/docs/v2/facets.md). For dashboards built from KPIs, trends, and breakdowns, use [Analytics](https://sigmie.com/docs/v2/analytics.md).

## Run aggregations

Pass a callback to `aggregate()`. Read each result from the response with `aggregation()` and a dot path that starts with the aggregation name:

```php
use Sigmie\Query\Aggs;

$response = $sigmie->newQuery('orders')
    ->matchAll()
    ->aggregate(function (Aggs $agg) {
        $agg->sum(name: 'turnover', field: 'price');
    })
    ->get();

$response->aggregation('turnover.value'); // 54.403
```

To return only the aggregations and no hits, set `size(0)`:

```php
$sigmie->newQuery('sales')
    ->matchAll()
    ->aggregate(function (Aggs $agg) {
        $agg->terms('top_products', 'product_id')->size(10);
    })
    ->size(0)
    ->get();
```

To aggregate over a subset of documents, replace `matchAll()` with a query. See [Advanced Queries](https://sigmie.com/docs/v2/query.md).

## Metric aggregations

A metric aggregation computes values over the documents in its scope.

| Method | Elasticsearch type | Result |
|--------|--------------------|--------|
| `sum(name, field)` | `sum` | `value` |
| `avg(name, field)` | `avg` | `value` |
| `min(name, field)` | `min` | `value` |
| `max(name, field)` | `max` | `value` |
| `stats(name, field)` | `stats` | `count`, `min`, `max`, `avg`, `sum` |
| `valueCount(name, field)` | `value_count` | `value`: the number of values, duplicates included |
| `cardinality(name, field)` | `cardinality` | `value`: the approximate number of distinct values |
| `percentiles(name, field, percents)` | `percentiles` | `values` keyed by percent. `percents` defaults to `[1, 5, 25, 50, 75, 95, 99]` |
| `percentileRanks(name, field, values)` | `percentile_ranks` | `values` keyed by value |
| `rate(name, field)` | `rate` | `value`. Only inside a date histogram |
| `topHits(name, size, sourceIncludes, sort, sourceExcludes)` | `top_hits` | `hits`. `size` defaults to 1 |
| `scriptedMetric(name, initScript, mapScript, combineScript, reduceScript, params)` | `scripted_metric` | `value` |

```php
$agg->stats(name: 'sales_stats', field: 'amount');

$response->aggregation('sales_stats');
// [
//     'count' => 133,
//     'min'   => 5.33,
//     'max'   => 128.58,
//     'avg'   => 73.53,
//     'sum'   => 9779.49,
// ]
```

`cardinality()` accepts `precisionThreshold()`. Below the threshold, counts are close to exact. Above it, they are estimates:

```php
$agg->cardinality(name: 'unique_users', field: 'user_id')->precisionThreshold(1000);
```

## Bucket aggregations

A bucket aggregation groups documents. Each bucket has a `key` and a `doc_count`, plus the results of its [sub-aggregations](#sub-aggregations).

| Method | Elasticsearch type |
|--------|--------------------|
| `terms(name, field)` | `terms` |
| `multiTerms(name, fields)` | `multi_terms` |
| `unionTerms(name, fields)` | `terms` with a script |
| `range(name, field, ranges)` | `range` |
| `histogram(name, field, interval, minDocCount, extendedBounds)` | `histogram` |
| `dateHistogram(name, field, interval, minDocCount, extendedBounds, format, timeZone)` | `date_histogram` |
| `autoDateHistogram(name, field, buckets, minimumInterval, timeZone)` | `auto_date_histogram` |
| `geoHashGrid(name, field, precision, size)` | `geohash_grid` |
| `significantText(name, field)` | `significant_text` |
| `composite(name, sources, size, after)` | `composite` |
| `missing(name, field)` | `missing` |
| `global(name)` | `global` |
| `nested(name, path, callable)` | `nested` |
| `filter(name, query)` | `filter` |
| `rangeFilter(name, field, ranges)` | `filter` with a `range` query |

### Terms

`terms()` groups documents by the values of a field. Use a `keyword` field, or a `text` field with a keyword sub-field.

```php
$agg->terms(name: 'category_terms', field: 'category')
    ->size(10)
    ->missing('N/A');

$response->aggregation('category_terms.buckets');
// [
//     ['key' => 'Fantasy', 'doc_count' => 20],
//     ['key' => 'Musical', 'doc_count' => 18],
//     ['key' => 'Adventure', 'doc_count' => 13],
//     ['key' => 'N/A', 'doc_count' => 7],
// ]
```

| Method | Effect |
|--------|--------|
| `size(int)` | Number of buckets to return. Elasticsearch defaults to 10. |
| `order(string $key, string $direction)` | Sort buckets by `_count`, `_key`, or a sub-aggregation name. |
| `missing(value)` | Put documents without the field into a bucket with this key. |
| `exclude(array\|string)` | Leave these values out. |

### Union terms

`unionTerms()` counts one label across several fields as one terms aggregation. Use it when one dimension lives in fields such as `champion_country` and `runner_up_country`:

```php
use Sigmie\Query\Aggregations\Metrics\Sum;
use Sigmie\Query\Aggs;

$agg->unionTerms('countries', ['champion_country', 'runner_up_country'])
    ->size(10)
    ->order('prize_total', 'desc')
    ->aggregate(fn (Aggs $sub): Sum => $sub->sum('prize_total', 'prize'));

$response->aggregation('countries.buckets');
// [
//     ['key' => 'Germany', 'doc_count' => 4, 'prize_total' => ['value' => 420.0]],
//     ...
// ]
```

A fixed Painless script reads the field names as parameters. A document with the same label in several of the fields counts once for that label. The [`unionBreakdown()`](https://sigmie.com/docs/v2/analytics.md#union-breakdown) widget adds mapping validation, time windows, filters, and normalized rows.

### Range

`range()` groups a numeric field into the ranges you give. `from` is inclusive and `to` is exclusive:

```php
$agg->range('price_ranges', 'price', [
    ['key' => '0-100', 'to' => 100],
    ['key' => '100-200', 'from' => 100, 'to' => 200],
    ['key' => '200+', 'from' => 200],
]);

$response->aggregation('price_ranges.buckets');
// [
//     ['key' => '0-100', 'to' => 100.0, 'doc_count' => 803],
//     ['key' => '100-200', 'from' => 100.0, 'to' => 200.0, 'doc_count' => 422],
//     ['key' => '200+', 'from' => 200.0, 'doc_count' => 343],
// ]
```

### Histogram

`histogram()` groups a numeric field into buckets of a fixed width. `minDocCount` defaults to `0`, so empty buckets are included:

```php
$agg->histogram(name: 'price_histogram', field: 'price', interval: 50);
```

### Date histogram

`dateHistogram()` groups a date field by time. A `CalendarInterval` case sends a calendar interval. A string such as `'15d'` sends a fixed interval:

```php
use Sigmie\Query\Aggregations\Enums\CalendarInterval;

$agg->dateHistogram(name: 'sales_over_time', field: 'created_at', interval: CalendarInterval::Month);
$agg->dateHistogram(name: 'fortnights', field: 'created_at', interval: '15d');
```

The `CalendarInterval` cases are `Minute`, `Hour`, `Day`, `Week`, `Month`, `Quarter`, and `Year`. `format` sets the format of each bucket's `key_as_string`, for example `'yyyy-MM'`.

### Auto date histogram

`autoDateHistogram()` lets Elasticsearch pick the interval that fits the number of buckets you ask for:

```php
$agg->autoDateHistogram(name: 'timeline', field: 'created_at', buckets: 12);
```

### Geohash grid

`geoHashGrid()` groups a `geo_point` field into geohash cells, for example to draw a map heatmap. `precision` goes from 1 to 12 and defaults to 5. A higher precision gives smaller cells. `size` caps the number of cells:

```php
$agg->geoHashGrid(name: 'areas', field: 'location', precision: 5);
```

## Sub-aggregations

Call `aggregate()` on a bucket aggregation to compute more aggregations inside each bucket:

```php
$agg->terms(name: 'category_terms', field: 'category')
    ->aggregate(function (Aggs $sub) {
        $sub->avg(name: 'avg_price', field: 'price');
        $sub->max(name: 'max_price', field: 'price');
    });

$response->aggregation('category_terms.buckets.0.avg_price.value');
```

Each category bucket holds `avg_price` and `max_price` next to `doc_count`.

## Filtered aggregations

`rangeFilter()` limits its sub-aggregations to documents in a range. Keys are range operators:

```php
$agg->rangeFilter('expensive_products', 'price', ['>=' => 100])
    ->aggregate(function (Aggs $sub) {
        $sub->terms(name: 'expensive_categories', field: 'category');
    });
```

`filter()` takes any query object, for example `new Term('status', 'active')` from `Sigmie\Query\Queries\Term\Term`.

## Pipeline aggregations

A pipeline aggregation reads the output of other aggregations through a `buckets_path`. Add one with `add()`:

```php
use Sigmie\Query\Aggregations\Enums\CalendarInterval;
use Sigmie\Query\Aggregations\Pipeline\AvgBucket;

$agg->dateHistogram('sales_per_month', 'created_at', CalendarInterval::Month)
    ->aggregate(function (Aggs $sub) {
        $sub->sum('total_sales', 'amount');
    });

$agg->add(new AvgBucket('avg_monthly_sales', 'sales_per_month>total_sales'));

$response->aggregation('avg_monthly_sales.value');
```

| Class in `Sigmie\Query\Aggregations\Pipeline` | Elasticsearch type | Place it |
|-----------------------------------------------|--------------------|----------|
| `AvgBucket(name, path)` | `avg_bucket` | Next to the bucket aggregation |
| `SumBucket(name, path)` | `sum_bucket` | Next to the bucket aggregation |
| `MinBucket(name, path)` | `min_bucket` | Next to the bucket aggregation |
| `MaxBucket(name, path)` | `max_bucket` | Next to the bucket aggregation |
| `CumulativeSum(name, path)` | `cumulative_sum` | Inside a histogram. Also `$sub->cumulativeSum(name, path)` |
| `SortBucket(name, path, order)` | `bucket_sort` | Inside the bucket aggregation |

`SortBucket` sorts the buckets of its parent by a sub-aggregation:

```php
use Sigmie\Query\Aggregations\Pipeline\SortBucket;

$agg->terms('products', 'product')
    ->aggregate(function (Aggs $sub) {
        $sub->sum('revenue', 'amount');
        $sub->add(new SortBucket('sort_by_revenue', 'revenue', 'desc'));
    });
```

## Missing values

`missing()` sets the value used for documents that do not have the field. Terms, date histogram, and auto date histogram aggregations accept it, and so do the `avg`, `min`, `max`, `sum`, `stats`, `cardinality`, `percentiles`, and `percentile_ranks` metrics:

```php
$agg->dateHistogram('histogram', 'date', CalendarInterval::Year)
    ->missing('2021-01-01');

$agg->max('count_max', 'count')->missing(0);
```


---

<!-- source: https://sigmie.com/docs/v2/magic-tags -->

# Magic Tags

Magic Tags adds a `keyword` field whose values an LLM writes: short labels that describe another field's content. The package prefers existing tags over new ones, so the tag list stays stable for search, filters, and agent tools.

> **Note:** Magic Tags is a separate [Sigmie package](https://sigmie.com/docs/v2/extending.md), not part of this repository. This page describes its intended behavior. The examples assume that you installed and registered the package.

## How the package stores tags

The package writes tags to two indices:

```
Main index documents                       Sidecar index (tag registry)
+---------------------------+              +-----------------------------+
| content (semantic text)   |              | magic_field_path (keyword)  |
| topic (magic_tags)        |    sync      | tag (short text + vectors)  |
| _embeddings.content ...   |  ─────────►  | _embeddings.tag ...         |
+---------------------------+              +-----------------------------+
        │                                            │
        │                                            │
   LLM + optional                          Same embedding API as
   classify-first                          the source field
```

The **main index** stores each document's tags as an array of strings. The mapping is `keyword` with `meta.type` set to `magic_tags`.

The **sidecar index** holds one row for each unique `(magic_field_path, tag)` pair, with an embedding of the tag text. Its name defaults to `{logicalName}__sigmie_magic_tags`. Each row has the `_id` `md5(path::tag)`, so a repeated write updates the row and does not add a duplicate. The sidecar uses the same embeddings API and dimensions as the source field.

The package runs as a `CollectionHook` during `merge()` and `add()`. `beforeBatch()` creates the sidecar index if it does not exist. `processBatch()` selects the tags for each document. `afterBatch()` upserts the new tags into the sidecar. For the hook lifecycle, see [Extending Sigmie](https://sigmie.com/docs/v2/extending.md).

## Install the package

Register the package on your `Sigmie` client:

```php
use Sigmie\Sigmie;
use Vendor\MagicTags\MagicTagsPackage;

$sigmie = new Sigmie($connection);
$sigmie->extend(new MagicTagsPackage());
```

`extend()` adds the `magicTags()` macro to `NewProperties` for the whole PHP process. It adds the collection hook to this `Sigmie` instance only.

## Add a magic tags field

The source field must be a [semantic](https://sigmie.com/docs/v2/semantic-search.md) text field, because the package reads its embeddings configuration to set up the sidecar.

```php
use Sigmie\Mappings\NewProperties;

$props = new NewProperties;

$props->text('content')
    ->semantic(api: 'my-embeddings', accuracy: 1, dimensions: 1024);

$props->magicTags('topic', fromField: 'content')
    ->api('my-llm');
```

Register both API names on the collection:

```php
$collection = $sigmie->collect('kb', refresh: true)
    ->properties($props)
    ->apis([
        'my-llm' => $llmApi,
        'my-embeddings' => $embeddingsApi,
    ]);
```

Index a document with `merge()` or `add()`:

```php
use Sigmie\Document\Document;

$collection->merge([
    new Document(['content' => 'How to reset a circuit breaker.']),
]);
```

The stored document now has a `topic` array of tags.

## Tune classification and deduplication

For each document, the package selects tags in this order:

1. **Classify.** This step runs only when `classifyFirst(true)` is set and the sidecar has at least `minTagsForClassification` tags. The package embeds the source text and compares it with a centroid for each tag, built from sample passages. Tags with a similarity of at least `classifyConfidence` apply without an LLM call.
2. **Generate.** If classification returns no tags, the LLM writes tags from the source text. The prompt lists existing tags so that the LLM can reuse them.
3. **Deduplicate.** The package compares each new tag with existing tags by embedding similarity. A tag above `similarityThreshold` counts as a duplicate.

Set these values on the field:

```php
$props->magicTags('topic', fromField: 'content')
    ->api('my-llm')
    ->embeddingsApi('my-embeddings')
    ->embeddingDimensions(1024)
    ->classifyFirst(true)
    ->minTagsForClassification(10)        // need 10+ tags before classifying
    ->classifyConfidence(0.3)             // minimum centroid similarity
    ->classifySamplesPerTag(5)            // passages per tag for centroid
    ->similarityThreshold(0.85)           // dedup threshold
    ->maxTags(5);
```

To always call the LLM, turn classification off:

```php
$props->magicTags('topic', fromField: 'content')
    ->api('my-llm')
    ->classifyFirst(false);
```

## Replace the LLM prompt

`prompt()` replaces the default LLM instructions:

```php
$props->magicTags('topic', fromField: 'content')
    ->api('my-llm')
    ->prompt(
        'You tag property-management support content. Return up to 5 lowercase '.
        'kebab-case tags. Prefer reusing tags from the existing list when applicable.'
    );
```

## Share one tag registry across indices

To use one sidecar for several indices, give each field the same `tagIndex()` name. The main index names stay different.

```php
$shared = 'property_app';

$kb = new NewProperties;
$kb->text('content')->semantic(api: 'my-embeddings', accuracy: 1, dimensions: 1024);
$kb->magicTags('topic', fromField: 'content')
    ->api('my-llm')
    ->tagIndex($shared);

$memory = new NewProperties;
$memory->text('content')->semantic(api: 'my-embeddings', accuracy: 1, dimensions: 1024);
$memory->magicTags('topic', fromField: 'content')
    ->api('my-llm')
    ->tagIndex($shared);
```

Both collections write to `property_app__sigmie_magic_tags`. The `magic_field_path` of each row shows which source field produced the tag.

> **Note:** The LLM prompt lists existing tags from the **current** main index only. To show the LLM tags from all collections, merge the tag lists yourself before you call `merge()`.

## Index documents that already have tags

To skip the pipeline for a batch whose documents already have their final tags, call `withoutHooks()`:

```php
$collection->withoutHooks()->merge($documents);
```

## List the tags for a filter or agent

To show the available tags in a filter menu or an agent tool, run a `terms` aggregation on the magic tags field:

```php
use Sigmie\Query\Aggs;
use Sigmie\Query\Queries\MatchAll;
use Sigmie\Query\Search as QuerySearch;

$aggs = new Aggs;
$aggs->terms('by_topic', 'topic')->size(20);

$response = (new QuerySearch($connection))
    ->index('kb')
    ->query(new MatchAll)
    ->aggs($aggs)
    ->size(0)
    ->get();

$buckets = $response->json('aggregations.by_topic.buckets');
// [['key' => 'returns', 'doc_count' => 42], ['key' => 'shipping', 'doc_count' => 18], ...]
```

This list is separate from the tag list that the package sends to the LLM, which is larger. For other bucket options, see [Aggregations](https://sigmie.com/docs/v2/aggregations.md).

## See also

- [Extending Sigmie](https://sigmie.com/docs/v2/extending.md): packages, macros, and collection hooks.
- [Semantic Search](https://sigmie.com/docs/v2/semantic-search.md): semantic fields and embeddings.
- [Mappings & Properties](https://sigmie.com/docs/v2/mappings.md): property builders.
- [Documents](https://sigmie.com/docs/v2/document.md): collections, `add()`, and `merge()`.


---

<!-- source: https://sigmie.com/docs/v2/analytics -->

# Analytics

`analytics()` builds a dashboard from named **widgets**, such as KPIs, trends, ranked lists, and funnels. `get()` runs every widget in one Elasticsearch request. For anything no widget covers, use [raw aggregations](https://sigmie.com/docs/v2/aggregations.md).

```php
use Sigmie\Analytics\Enums\Metric;
use Sigmie\Query\Aggregations\Enums\CalendarInterval;

$dashboard = $orders->analytics('created_at')
    ->from(new DateTimeImmutable('-30 days'))
    ->to(new DateTimeImmutable)
    ->filters("status:'paid'")
    ->kpiDelta('revenue', Metric::Sum, 'amount')
    ->trend('revenue_over_time', Metric::Sum, 'amount', CalendarInterval::Day)
    ->breakdown('top_products', 'product', Metric::Sum, 'amount', limit: 5)
    ->get();
```

`analytics()` is a method of every [`SigmieIndex`](https://sigmie.com/docs/v2/mappings.md). Its argument is the **timeline**, the date field that every widget filters and buckets by. The window defaults to the last 30 days.

`$sigmie->analytics($indexName, $dateField)` starts the same builder without an index class. Call `properties()` on it so that filters are typed and text fields group on their `.keyword` sub-field.

## Metrics

Widgets that measure something take a `Metric`:

| Metric           | Value                                                    |
| ---------------- | -------------------------------------------------------- |
| `Metric::Sum`    | Total of a numeric field                                 |
| `Metric::Avg`    | Average of a numeric field                               |
| `Metric::Min`    | Minimum value                                            |
| `Metric::Max`    | Maximum value                                            |
| `Metric::Count`  | Number of documents with a value in the field            |
| `Metric::Unique` | Distinct values. Approximate above 40,000 distinct values |
| `Metric::Median` | 50th percentile                                          |

When you give `Metric::Count` no field, it counts the timeline field, so it counts every document in the window.

## Widgets

`get()` returns an array keyed by widget name. Each value has a `type` key and the chart data. The outputs below are shortened.

Every widget method also takes [`filter:`](#per-widget-filter) and [`window:`](#per-widget-window) as its last arguments.

### KPI

`kpi()` returns one number for the window.

```php
$orders->analytics('created_at')
    ->kpi('revenue', Metric::Sum, 'amount')
    ->kpi('customers', Metric::Unique, 'customer_id')
    ->get();

// ['revenue' => ['type' => 'kpi', 'value' => 48210.0, 'count' => 1240, ...], ...]
```

### KPI delta

`kpiDelta()` compares the window with the previous window of equal length. [Calendar periods](#named-ranges) compare with the previous calendar period. `change_pct` is `null` when the previous value is `0` or missing.

```php
$orders->analytics('created_at')
    ->from($monthStart)->to($monthEnd)
    ->kpiDelta('revenue', Metric::Sum, 'amount')
    ->get();

// ['revenue' => ['type' => 'kpi_delta', 'value' => 300.0, 'previous' => 150.0, 'change_pct' => 100.0, ...]]
```

### Trend

`trend()` returns a metric per time bucket. To change from daily to monthly buckets, change only the `CalendarInterval`. The series spans the whole window, including empty buckets.

```php
$orders->analytics('created_at')
    ->trend('revenue', Metric::Sum, 'amount', CalendarInterval::Day)
    ->get();

// ['revenue' => ['type' => 'trend', 'interval' => 'Day', 'series' => [
//     ['label' => '2024-01-01T00:00:00.000Z', 'value' => 150.0], ...
// ]]]
```

Intervals: `Minute`, `Hour`, `Day`, `Week`, `Month`, `Quarter`, and `Year`.

### Grouped trend

`groupedTrend()` returns one series for each of the top `limit` values of `groupBy`, such as revenue by product over time.

```php
$orders->analytics('created_at')
    ->groupedTrend('by_product', Metric::Sum, 'amount', groupBy: 'product', interval: CalendarInterval::Day, limit: 5)
    ->get();

// ['by_product' => ['type' => 'grouped_trend', 'groups' => [
//     ['group' => 'A', 'series' => [['label' => ..., 'value' => 370.0], ...]], ...
// ]]]
```

The `field` argument is required. To count events, pass an empty string:

```php
$events->analytics('occurred_at')
    ->groupedTrend(
        'events_by_type',
        Metric::Count,
        field: '',
        groupBy: 'event_type',
        interval: CalendarInterval::Day,
    )
    ->get();
```

### Breakdown

`breakdown()` returns the top `limit` values of a field, ranked by a metric. `direction` defaults to `'desc'`. Pass `'asc'` to return the lowest values first.

```php
$orders->analytics('created_at')
    ->breakdown('top_products', 'product', Metric::Sum, 'amount', limit: 10)
    ->breakdown('smallest_queues', 'queue', Metric::Count, limit: 5, direction: 'asc')
    ->get();

// ['top_products' => ['type' => 'breakdown', 'rows' => [
//     ['key' => 'A', 'value' => 370.0, 'count' => 3, ...], ...
// ]], ...]
```

### Multi breakdown

`multiBreakdown()` ranks combinations of fields, such as product and channel pairs.

```php
$orders->analytics('created_at')
    ->multiBreakdown(
        'top_product_channels',
        ['product', 'channel'],
        Metric::Sum,
        field: 'amount',
        limit: 10,
    )
    ->get();

// ['top_product_channels' => ['type' => 'multi_breakdown', 'rows' => [
//     ['key' => ['product' => 'A', 'channel' => 'online'],
//      'key_values' => ['A', 'online'], 'label' => 'A / online',
//      'value' => 240.0, 'count' => 2], ...
// ]]]
```

### Union breakdown

`unionBreakdown()` ranks labels across several fields. A country in `champion_country` or in `runner_up_country` counts toward one country row.

```php
$tournaments->analytics('played_at')
    ->unionBreakdown(
        'country_appearances',
        ['champion_country', 'runner_up_country'],
        Metric::Count,
        limit: 10,
    )
    ->unionBreakdown(
        'country_prize',
        ['champion_country', 'runner_up_country'],
        Metric::Sum,
        field: 'prize',
        limit: 10,
    )
    ->get();

// ['country_appearances' => ['type' => 'union_breakdown', 'rows' => [
//     ['key' => 'Germany', 'value' => 4, 'count' => 4, ...], ...
// ]], ...]
```

The fields need compatible, aggregatable mapping types. A document with the same label in two fields counts once. `limit` and `direction` work as in `breakdown()`. See [union terms](https://sigmie.com/docs/v2/aggregations.md#union-terms) for the aggregation.

### Grouped metrics

`groupedMetrics()` returns several metrics per group and ranks groups by the metric whose `key` is `sortMetric`. Metrics other than count need a numeric `field`. `minCount` drops smaller groups.

```php
$orders->analytics('created_at')
    ->groupedMetrics(
        'product_metrics',
        groupBy: 'product',
        metrics: [
            ['key' => 'orders', 'label' => 'Orders', 'metric' => Metric::Count],
            ['key' => 'average_amount', 'label' => 'Average amount', 'metric' => Metric::Avg, 'field' => 'amount'],
        ],
        sortMetric: 'orders',
        limit: 10,
        minCount: 2,
    )
    ->get();

// ['product_metrics' => ['type' => 'grouped_metrics', 'rows' => [
//     ['key' => 'A', 'value' => 3, 'count' => 3,
//      'metrics' => ['orders' => 3, 'average_amount' => 123.33],
//      'metric_populations' => [
//          'orders' => ['document_count' => 3, 'value_count' => 3, 'field' => ''],
//          'average_amount' => ['document_count' => 3, 'value_count' => 3, 'field' => 'amount'],
//      ]], ...
// ]]]
```

In `metric_populations`, `value_count` below `document_count` means some documents in the group have no value for that metric.

### Distribution

`distribution()` counts documents in fixed-width ranges of a numeric field.

```php
$orders->analytics('created_at')
    ->distribution('order_sizes', 'amount', interval: 100)
    ->get();

// ['order_sizes' => ['type' => 'distribution', 'buckets' => [['label' => 0, 'count' => 3], ['label' => 100, 'count' => 1], ...]]]
```

### Histogram metric

`histogramMetric()` computes a metric in each fixed-width range of `bucketField`, such as the average rating per price band.

```php
$products->analytics('created_at')
    ->histogramMetric(
        'average_rating_by_price',
        bucketField: 'price',
        interval: 50,
        metric: Metric::Avg,
        field: 'rating',
    )
    ->get();

// ['average_rating_by_price' => [
//     'type' => 'histogram_metric',
//     'bucket_field' => 'price',
//     'interval' => 50,
//     'series' => [
//         ['label' => 0, 'value' => 4.2, 'count' => 18],
//         ['label' => 50, 'value' => 4.6, 'count' => 7], ...
//     ],
// ]]
```

Each `label` is the lower bound of its range. `50` covers 50 up to, but not including, 100.

### Percentiles

`percentiles()` returns the given percentiles of a numeric field. The default is `[50, 75, 95, 99]`.

```php
$requests->analytics('created_at')
    ->percentiles('latency', 'duration_ms', [50, 95, 99])
    ->get();

// ['latency' => ['type' => 'percentiles', 'percentiles' => ['50' => 120.0, '95' => 880.0, '99' => 1450.0]]]
```

### Cumulative

`cumulative()` returns a running total per time bucket, from a `cumulative_sum` pipeline aggregation.

```php
$orders->analytics('created_at')
    ->cumulative('growth', Metric::Sum, 'amount', CalendarInterval::Day)
    ->get();

// ['growth' => ['type' => 'cumulative', 'series' => [
//     ['label' => '2024-01-01...', 'value' => 150.0],
//     ['label' => '2024-01-02...', 'value' => 350.0], ...
// ]]]
```

### Stats

`stats()` returns `count`, `min`, `max`, `avg`, and `sum` of a numeric field.

```php
$orders->analytics('created_at')
    ->stats('order_size', 'amount')
    ->get();

// ['order_size' => ['type' => 'stats', 'count' => 5, 'min' => 30.0, 'max' => 200.0, 'avg' => 90.0, 'sum' => 450.0]]
```

### Table

`table()` returns matching documents instead of a number. It uses a `top_hits` aggregation, so filters and windows apply as for other widgets.

| Argument | Default  | Effect                                                              |
| -------- | -------- | ------------------------------------------------------------------- |
| `fields` | `[]`     | Source fields to return. An empty array returns the whole document. |
| `limit`  | `10`     | Number of rows.                                                     |
| `sort`   | `null`   | A `"field:direction"` string. The direction defaults to `desc`.     |
| `except` | `[]`     | Source fields to remove, even when `fields` lists their parent.     |

```php
$orders->analytics('created_at')
    ->table('recent', fields: ['amount', 'product'], limit: 20, sort: 'amount:desc')
    ->get();

// ['recent' => ['type' => 'table', 'rows' => [
//     ['id' => 'ord_8f2c…', 'document' => ['amount' => 200, 'product' => 'A']], ...
// ]]]

$customers->analytics('created_at')
    ->table('recent', fields: ['name', 'contacts'], except: ['contacts.email'])
    ->get();
```

`table()` does not paginate. For a paginated table, use [`search()`](https://sigmie.com/docs/v2/search.md) in the same request through [multi-search](#batch-with-multi-search).

### Funnel

`funnel()` counts the documents that match each step. `steps` is an ordered array of `label => slice`, where a slice is a [filter-parser](https://sigmie.com/docs/v2/filter-parser.md) string or a query object. `conversion` compares each step with the first step, and `step_conversion` with the previous step.

```php
$events->analytics('created_at')
    ->funnel('signup', [
        'visited' => "event:'visit'",
        'signed'  => "event:'signup'",
        'paid'    => "event:'paid'",
    ])
    ->get();

// ['signup' => ['type' => 'funnel', 'steps' => [
//     ['label' => 'visited', 'count' => 4, 'conversion' => 1.0,  'step_conversion' => 1.0],
//     ['label' => 'signed',  'count' => 2, 'conversion' => 0.5,  'step_conversion' => 0.5],
//     ['label' => 'paid',    'count' => 1, 'conversion' => 0.25, 'step_conversion' => 0.5],
// ]]]
```

### Heatmap

`heatmap()` returns a `rowField` by `colField` matrix. Each cell holds a metric, by default a document count. `rowLimit` and `colLimit` default to `10`.

```php
$orders->analytics('created_at')
    ->heatmap('by_region', rowField: 'region', colField: 'device', metric: Metric::Sum, field: 'amount')
    ->get();

// ['by_region' => ['type' => 'heatmap', 'rows' => [
//     ['key' => 'US', 'count' => 3, 'cells' => [
//         ['key' => 'mobile', 'value' => 220.0, 'count' => 2],
//         ['key' => 'desktop', 'value' => 90.0, 'count' => 1],
//     ]], ...
// ]]]
```

### Retention

`retention()` returns a cohort table. It groups entities by the period of their `cohortField` date, then counts distinct `idField` values in each later timeline period. `size` is the cohort's distinct entity count.

```php
$signups->analytics('active_at')
    ->retention('cohorts', cohortField: 'signup_at', idField: 'user_id', interval: CalendarInterval::Week)
    ->get();

// ['cohorts' => ['type' => 'retention', 'cohorts' => [
//     ['cohort' => '2024-01-01...', 'size' => 2, 'periods' => [
//         ['label' => '2024-01-01...', 'value' => 2],
//         ['label' => '2024-01-08...', 'value' => 1],
//     ]], ...
// ]]]
```

### Geo

`geo()` groups documents into geohash cells of a `geo_point` field. A higher `precision` gives smaller cells. Each cell holds a document count unless you pass `metric:` and `metricField:`.

```php
$orders->analytics('created_at')
    ->geo('areas', 'location', precision: 5)
    ->get();

// ['areas' => ['type' => 'geo', 'precision' => 5, 'buckets' => [
//     ['geohash' => 'u173z', 'value' => 2, 'count' => 2], ...
// ]]]
```

## Filters

`filters()` applies a [filter-parser](https://sigmie.com/docs/v2/filter-parser.md) string to every widget. `filterQuery()` adds a query object with AND, like [`NewSearch::filterQuery()`](https://sigmie.com/docs/v2/search.md).

```php
use Sigmie\Query\Queries\Term\Range;

$orders->analytics('created_at')
    ->filters("status:'paid'")
    ->filterQuery(new Range('amount', ['>=' => 100]))
    ->kpi('big_orders', Metric::Sum, 'amount')
    ->get();
```

### Per-widget filter

The `filter:` argument limits one widget to a slice of documents, in addition to the dashboard filters. It takes a filter-parser string or a query object. On `kpiDelta()`, it applies to both windows.

```php
use Sigmie\Query\Queries\Term\Term;

$orders->analytics('created_at')
    ->from($start)->to($end)
    ->kpi('total', Metric::Unique, 'customer_id')
    ->kpi('engaged', Metric::Unique, 'customer_id', filter: "status:'opened' OR status:'completed'")
    ->kpi('completed', Metric::Unique, 'customer_id', filter: new Term('status', 'completed'))
    ->trend('paid_revenue', Metric::Sum, 'amount', CalendarInterval::Day, filter: "status:'paid'")
    ->kpiDelta('refunds', Metric::Count, filter: "status:'refunded'")
    ->get();
```

## Time window

`from()` and `to()` set the dashboard window. It includes `from` and excludes `to`. Create both dates in the user's timezone to align the window with the user's day.

### Per-widget window

The `window:` argument gives one widget its own time range. It takes a [`Period`](#named-ranges) or a `[$from, $to]` pair.

```php
use Sigmie\Analytics\Enums\Period;

$orders->analytics('created_at')
    ->from($last30Start)->to($now)
    ->kpi('lifetime_revenue', Metric::Sum, 'amount', window: [$accountStart, $now])
    ->kpi('this_year', Metric::Sum, 'amount', window: Period::ThisYear)
    ->trend('recent_revenue', Metric::Sum, 'amount', CalendarInterval::Day)
    ->get();
```

A `kpiDelta()` with its own `window:` always compares against the previous window of equal length.

## Named ranges

`range()` sets the window to a `Period`, converted to fixed dates in the configured timezone. Weeks start on Monday.

```php
use Sigmie\Analytics\Enums\Period;

$orders->analytics('created_at')->timezoneOffset(540)->range(Period::Last7Days);
```

Periods: `Today`, `Yesterday`, `ThisWeek`, `LastWeek`, `ThisMonth`, `LastMonth`, `ThisQuarter`, `LastQuarter`, `ThisYear`, `LastYear`, `Last7Days`, `Last30Days`, and `Last90Days`.

With a week, month, quarter, or year period, `kpiDelta()` compares with the previous calendar period: `ThisMonth` compares with last month. Other periods compare with the previous window of equal length.

> **Note:** `range()` resolves the period when you call it. Set the timezone first.

## Timezone

Analytics uses UTC by default, so a daily chart puts a Tokyo sale made just after midnight into the previous day. Set a timezone to align time buckets and named ranges with local time:

```php
$orders->analytics('created_at')
    ->timezoneOffset(540)
    ->range(Period::ThisMonth)
    ->trend('sales', Metric::Sum, 'amount', CalendarInterval::Day)
    ->get();
```

| Method                       | Argument                                                                               |
| ---------------------------- | -------------------------------------------------------------------------------------- |
| `timezoneOffset(int)`        | Minutes east of UTC: Tokyo is `540`, New York is `-300`, India is `330`, Nepal is `345`. |
| `timezone(string)`           | An IANA name such as `'Asia/Tokyo'`, or a fixed offset such as `'+09:00'`.              |

A fixed offset ignores daylight saving time. An IANA name follows it. In a browser, `-new Date().getTimezoneOffset()` gives the value for `timezoneOffset()`.

## Hits alongside widgets

`getWithHits()` returns the widgets and the matching documents from one request. The request has `size(0)` by default, so configure the hits with `search()`.

```php
$result = $orders->analytics('created_at')
    ->from($start)->to($end)
    ->kpi('revenue', Metric::Sum, 'amount')
    ->search(function ($search) {
        $search
            ->size(20)
            ->sortString('amount:desc')
            ->postFilterString("status:'paid'")
            ->trackTotalHits(true);
    })
    ->getWithHits();

$result['widgets']['revenue']; // normalized KPI result
$result['hits'];               // raw Elasticsearch hits block
```

The result also has `took` and `timed_out`. A `post_filter` limits the hits but not the widgets, so the KPI above counts all orders and the hits show paid orders.

## Batch with multi-search

When the rows need a different query, `newAnalytics()` adds the dashboard to a [multi-search](https://sigmie.com/docs/v2/search.md). Its result is the widget array that `get()` returns.

```php
$multi = $sigmie->newMultiSearch();

$multi->newAnalytics($orders, 'created_at', 'metrics')
    ->from($start)->to($end)
    ->kpiDelta('revenue', Metric::Sum, 'amount')
    ->trend('revenue_over_time', Metric::Sum, 'amount', CalendarInterval::Day);

$multi->newSearch($orders->name(), 'rows')
    ->queryString('')->filters("status:'paid'")->page(3, 20);

[$metrics, $rows] = $multi->get();
```

For an existing `Analytics` instance, pass `toSearch()` to `addQuery()` and convert its response with `formatResponse()`:

```php
$multi->addQuery($analytics->toSearch(), 'metrics');
[$metrics, $rows] = $multi->get();
$dashboard = $analytics->formatResponse($metrics);
```

## Analytics tool for AI agents

An index that uses the [`AsTool`](https://sigmie.com/docs/v2/laravel-ai.md) trait includes an `analytics` tool in `tools()`. The agent selects a `widget`, such as `trend`, and passes its arguments.

```php
class OrderIndex extends SigmieIndex
{
    use AsTool;
    // ...
}

// In your agent:
public function tools(): array
{
    return [
        ...app(OrderIndex::class)->tools("user_id:{$this->user->id}"),
    ];
}
```

Tool arguments that differ from the PHP API:

| Argument                                            | Meaning                                                                    |
| --------------------------------------------------- | -------------------------------------------------------------------------- |
| `date_field`                                        | The timeline. Must be a date field of the index.                           |
| `range`, `timezone_offset`                          | A period name such as `this_month` or `last_7_days`, and minutes east of UTC. |
| `field: null`                                       | A count with no field, for example on `grouped_trend`.                     |
| `sort`                                              | `"metric:asc"` or `"metric:desc"` on ranked widgets. `"field:direction"` on `table`. |
| `group_by_fields`                                   | Two or more compatible fields for `multi_breakdown` and `union_breakdown`. |
| `include_hits`, `hit_fields`, `hit_sort`, `hit_limit`, `hit_filters` | Return the matching documents with the widget.                |

Your code sets the `$baseFilter` argument of `tools()`, never the agent. Sigmie adds it with AND to every query, so the agent sees only that slice.


---

<!-- source: https://sigmie.com/docs/v2/facets -->

# Facets

Facets are the counts and ranges beside search results, such as "Brand: Apple (12), Dell (8)". `facets()` on a search builds them from your property definitions and returns them under the response's `facets` key.

```php
use Sigmie\Mappings\NewProperties;

$props = new NewProperties;
$props->category('brand');
$props->price();

$response = $sigmie->newSearch('products')
    ->properties($props)
    ->queryString('laptop')
    ->facets('brand price:100')
    ->get();

$response->json('facets');
// ['brand' => ['Apple' => 5, 'Dell' => 3], 'price' => ['min' => 299, 'max' => 1499, 'histogram' => [...]]]
```

With an empty query string, the facets cover every document that the filters allow.

## The facets() method

```php
facets(string $facets, string $filters = '', bool $throwOnError = false)
```

| Parameter | Description |
|-----------|-------------|
| `$facets` | Space-separated list of fields to facet. See [Facet string](#facet-string). |
| `$filters` | The values the user selected. See [Facet filter string](#facet-filter-string). |
| `$throwOnError` | `false` drops an invalid clause and records it under the response's `errors` key. `true` throws a `ParseException`. |

```php
->facets('brand category', "brand:'Apple'", throwOnError: true)
```

## Facet string

Each entry is a field path with an optional parameter after `:`.

| Entry | Field types | Meaning |
|-------|-------------|---------|
| `brand` | All | Facet with default parameters. |
| `brand:20` | Term fields | Return the top 20 values. Default 10. |
| `brand:20,asc` | Term fields | Return the top 20 values, sorted by value. Also `desc`. |
| `price:100` | `price` | Histogram bucket width of 100. Default 10. |
| `attributes.color` | Nested | Facet a field inside a nested object. |

`number`, `date`, and `datetime` facets take no parameter.

## Facet filter string

The facet filter string holds the values the user selected. It uses [filter parser](https://sigmie.com/docs/v2/filter-parser.md) clauses separated by spaces:

```php
->facets('color size', "color:'red' color:'blue' size:'lg' price:100..200")
```

Sigmie groups the clauses by field. It joins the clauses of one field with the field's [facet logic](#facet-logic), then joins the fields with `AND`. When `color` is disjunctive, the string above becomes:

```
(color:'red' OR color:'blue') AND (size:'lg') AND (price:100..200)
```

When `color` is conjunctive, the first group is `(color:'red' AND color:'blue')`.

The facet filter string does not accept `AND`, `OR`, `AND NOT`, or parentheses. If it contains one, Sigmie drops the whole facet filter string and records the error, or throws when `$throwOnError` is `true`.

### How facet filters apply

The facet filter narrows the hits. For the counts, each facet ignores the clauses on its own field and applies the clauses on every other field. A user who selects `color:'green'` still sees every color with its count, and the size facet shows only the sizes that exist in green:

```php
$sigmie->newSearch('products')
    ->properties($props)
    ->filters('stock>0')
    ->facets('color size', "color:'green'")
    ->get();
```

| | `filters()` | Facet filter string |
|-|-------------|---------------------|
| Narrows the hits | Yes | Yes |
| Narrows the counts of the same field | Yes | No |
| Narrows the counts of other fields | Yes | Yes |
| Accepts `AND`, `OR`, parentheses | Yes | No |

Use `filters()` for constraints the user does not pick in the sidebar, such as `stock>0`. See [Search](https://sigmie.com/docs/v2/search.md#filter).

## Facet logic

Facet logic sets how several selected values of one field combine. Set it on the property:

```php
$props->category('color')->facetDisjunctive();
$props->category('material')->facetConjunctive();
```

| Logic | Values of one field | Use for |
|-------|---------------------|---------|
| Conjunctive (default) | Joined with `AND`. A hit must have every selected value. | Multi-value fields, such as tags. |
| Disjunctive | Joined with `OR`. A hit must have one of the selected values. | Single-value fields, such as brand or size. |

Different fields always combine with `AND`.

## Results by field type

| Field type | Facet result |
|------------|--------------|
| `keyword`, `category`, `tags`, `caseSensitiveKeyword` | Value counts |
| `text()->keyword()` | Value counts of the keyword sub-field |
| `number` | `count`, `min`, `max`, `avg`, `sum` |
| `price` | `min`, `max`, `histogram` |
| `date`, `datetime` | `count`, `min`, `max` as date strings |

A `text` field without `->keyword()` cannot be faceted.

### Term fields

`keyword`, `category`, and `tags` fields return each value with its count:

```php
$props->category('brand');
$props->keyword('color');

$response = $sigmie->newSearch('products')
    ->properties($props)
    ->queryString('shoes')
    ->facets('brand color')
    ->get();

$response->json('facets');
// ['brand' => ['Nike' => 15, 'Adidas' => 12], 'color' => ['black' => 10, ...]]
```

A `caseSensitiveKeyword` field counts each spelling on its own:

```php
$props->caseSensitiveKeyword('tag');

$response->facet('tag');
// ['sport' => 2, 'Sport' => 1]
```

A `text` field with a keyword sub-field returns the counts of that sub-field:

```php
$props->text('author')->keyword();

$response->facet('author');
// ['Jane Doe' => 14, 'John Smith' => 9]
```

### Price fields

A `price` field returns the lowest price, the highest price, and a histogram. Each histogram key is the lower bound of a bucket:

```php
$props->price();

$response = $sigmie->newSearch('products')
    ->properties($props)
    ->queryString('laptop')
    ->facets('price:100')
    ->get();

$response->facet('price');
// [
//     'min' => 299,
//     'max' => 1499,
//     'histogram' => [
//         200 => 3,    // 3 products from 200 to 299.99
//         300 => 8,
//         400 => 5,
//         ...
//     ],
// ]
```

### Number fields

A `number` field returns statistics:

```php
$props->number('rating');

$response->facet('rating');
// [
//     'count' => 127,
//     'min' => 1.0,
//     'max' => 5.0,
//     'avg' => 4.3,
//     'sum' => 546.1,
// ]
```

### Date fields

A `date` or `datetime` field returns the period it covers: the document `count`, the earliest date, and the latest date.

```php
$props->date('published_at');

$response = $sigmie->newSearch('posts')
    ->properties($props)
    ->queryString('')
    ->facets('published_at')
    ->get();

$response->facet('published_at');
// [
//     'count' => 3,
//     'min' => '2023-01-04T00:00:00.000Z',
//     'max' => '2025-11-30T00:00:00.000Z',
// ]
```

`min` and `max` are date strings in the field's first format, not epoch milliseconds. When no document matches, `count` is `0` and `min` and `max` are `null`. Facet filters on other fields narrow the range:

```php
$response = $sigmie->newSearch('posts')
    ->properties($props)
    ->queryString('')
    ->facets('published_at', "author:'Ada'")
    ->get();

$response->facet('published_at');
// ['count' => 2, 'min' => '2023-01-04T00:00:00.000Z', 'max' => '2024-06-18T00:00:00.000Z']
```

A date facet is a range, not a list of values. For counts per day, month, or year, use the [analytics](https://sigmie.com/docs/v2/analytics.md) `trend` widget or a `dateHistogram` [aggregation](https://sigmie.com/docs/v2/aggregations.md).

## Nested fields

Facet a field inside a `nested` property with its dot path. The result has the same shape as a root field of the same type:

```php
$props->nested('attributes', function (NewProperties $p) {
    $p->keyword('color');
    $p->price();
});

$response = $sigmie->newSearch('products')
    ->properties($props)
    ->queryString('shirt')
    ->facets('attributes.color attributes.price:50')
    ->get();

$response->facet('attributes.color');
// ['red' => 12, 'blue' => 7]

$response->facet('attributes.price');
// ['min' => 20, 'max' => 180, 'histogram' => [0 => 4, 50 => 9, 100 => 6, 150 => 2]]
```

Nesting can go deeper, for example `product.variants.size`.

## Read facet results

`facet()` returns the facet of one field, root or nested. It returns `null` for a field that is not in the properties or cannot be faceted.

```php
$response->facet('price');            // ['min' => ..., 'max' => ..., 'histogram' => [...]]
$response->facet('rating');           // ['count' => ..., 'min' => ..., 'max' => ..., 'avg' => ..., 'sum' => ...]
$response->facet('attributes.color'); // ['red' => 12, 'blue' => 7]
```

`json('facets')` returns every requested facet, keyed by field path:

```php
$response->json('facets');       // ['brand' => [...], 'attributes.color' => [...]]
$response->json('facets.brand');
```

> **Note:** `json()` reads `.` as a path separator, so `json('facets.attributes.color')` does not find a nested facet. Use `facet('attributes.color')`.


---

<!-- source: https://sigmie.com/docs/v2/recommendations -->

# Recommendations

`newRecommend()` returns documents similar to one or more documents you already have, the **seed documents**. Use it for "You might also like", "Related articles", and "Customers who viewed this also viewed" lists.

It searches with the vectors stored on the seeds, so it calls no embeddings API. When the user types a query, use [semantic search](https://sigmie.com/docs/v2/semantic-search.md) instead.

## Before you start

You need an index with [semantic fields](https://sigmie.com/docs/v2/semantic-search.md#define-semantic-fields). Index the documents with embeddings, which is the default. A collection with `populateEmbeddings(false)` stores no vectors, and its documents cannot be seeds.

The examples on this page use these properties:

```php
use Sigmie\Mappings\NewProperties;

$props = new NewProperties;
$props->text('name')->semantic(api: 'embeddings', dimensions: 384);
$props->text('category')->semantic(api: 'embeddings', dimensions: 384);
$props->text('brand')->semantic(api: 'embeddings', dimensions: 384);
$props->text('description')->semantic(api: 'embeddings', dimensions: 384);
$props->number('price');
$props->number('rating');
$props->bool('in_stock');
```

## Recommend from one document

Pass the properties, the seed ID, and at least one semantic field, then call `hits()`:

```php
$hits = $sigmie->newRecommend('products')
    ->properties($props)
    ->seedIds(['product-42'])
    ->field('category')
    ->field('name')
    ->topK(5)
    ->hits();

foreach ($hits as $hit) {
    echo "{$hit->_id}: {$hit['name']}\n";
}
```

`hits()` returns up to `topK` hits, best first. `topK` defaults to 10. The seed documents never appear in the results. `hits()` returns an empty array when no seed ID exists in the index or when you call no `field()`.

`seedIds()` fetches the seed documents when you call it.

## Weight the fields

Each `field()` call adds one semantic field and its weight. The default weight is `1.0`. A field with a higher weight has more influence on the ranking:

```php
$sigmie->newRecommend('products')
    ->properties($props)
    ->seedIds(['product-42'])
    ->field('category', weight: 3.0)
    ->field('brand', weight: 2.0)
    ->field('description', weight: 1.0)
    ->topK(10)
    ->hits();
```

Pass only semantic fields. `field()` reads the vectors that the seed stores for that field, and a non-semantic field has none.

## Recommend from several documents

Pass several seed IDs, for example the items in a cart or the user's recent views:

```php
$sigmie->newRecommend('products')
    ->properties($props)
    ->seedIds(['product-123', 'product-456'])
    ->field('category', weight: 2.0)
    ->field('name')
    ->hits();
```

Sigmie runs one vector search for each seed and fetches `topK × 10` candidates from each. It merges the lists with Reciprocal Rank Fusion (RRF). Each list adds `1 / (k + rank)` to the score of every document in it, so a document that ranks well for several seeds rises to the top. A document at rank 1 for one seed and rank 3 for another scores `1/61 + 1/63` with the default `k` of 60.

To change `k`, call `rrf()`. A higher `k` gives lower-ranked candidates more influence:

```php
->rrf(rrfRankConstant: 80)
```

## Filter the candidates

To limit the candidates, pass a [filter parser](https://sigmie.com/docs/v2/filter-parser.md) string to `filter()`:

```php
$sigmie->newRecommend('products')
    ->properties($props)
    ->seedIds(['macbook-pro-16-2023'])
    ->field('category', weight: 3.0)
    ->field('brand', weight: 2.0)
    ->filter('in_stock:true AND price<=2000 AND rating>=4')
    ->topK(10)
    ->hits();
```

## Diversify the results

Without diversity, ten results can be ten near-copies of the seed, such as one sneaker in ten colors. To spread the results, call `mmr()`. It turns on Maximal Marginal Relevance (MMR):

```php
$sigmie->newRecommend('products')
    ->properties($props)
    ->seedIds(['blue-nike-running-shoe'])
    ->field('category', weight: 2.0)
    ->field('name')
    ->mmr(lambda: 0.5)
    ->topK(10)
    ->hits();
```

MMR picks results one at a time. Each pick has the highest score for:

```
lambda × similarity_to_seeds − (1 − lambda) × similarity_to_already_picked
```

| `lambda` | Effect |
|----------|--------|
| `1.0` | Relevance only, no diversity |
| `0.5` | Default. Balanced |
| `0.0` | Diversity only |

Sigmie runs MMR on each field, then merges the per-field lists with RRF, weighted by the field weights.

> **Note:** MMR compares every candidate with every picked result, and the pool holds `topK × 10` candidates. A large `topK` makes it slow. Filters make the pool smaller.

Skip MMR when you need the closest matches, for example for duplicate detection.

## Troubleshoot

**No results.** Check that the seed IDs exist and that the filter does not exclude every candidate. Check that the seeds were indexed with embeddings.

**Poor matches.** Check that every field you pass to `field()` is semantic. Raise the weight of the field that best describes similarity in your data, such as `category`.

**Slow requests.** Lower `topK`, add filters, or turn off MMR.


---

<!-- source: https://sigmie.com/docs/v2/analysis -->

# Text Analysis

Analysis turns a string into the tokens that Elasticsearch stores and compares. It runs twice: on each text field when you index a document, and on the query string when you search. A query matches a document when both sides produce the same tokens. That is why Sigmie runs one analyzer on both sides.

## An analyzer has three stages

```
input text
   │
   ▼
[Char filters]    change the raw string (strip HTML, map symbols)
   │
   ▼
[Tokenizer]       split the string into tokens
   │
   ▼
[Token filters]   change, add, or remove tokens (lowercase, stopwords)
   │
   ▼
tokens
```

An analyzer has exactly one tokenizer. Char filters and token filters are optional, and an analyzer can have many of each.

| Stage | Works on | Example | Reference |
|-------|----------|---------|-----------|
| Char filters | The whole string, before it is split | Strip HTML tags | [Character Filters](https://sigmie.com/docs/v2/char-filters.md) |
| Tokenizer | The whole string, once | Split on whitespace | [Tokenizers](https://sigmie.com/docs/v2/tokenizers.md) |
| Token filters | Each token, one filter after another | Lowercase every token | [Token Filters](https://sigmie.com/docs/v2/token-filters.md) |

## Follow one string through the pipeline

This index builds an analyzer with one stage of each kind:

```php
$sigmie->newIndex('quotes')
    ->stripHTML()
    ->tokenizeOnWhitespaces()
    ->lowercase()
    ->stopwords(['are', 'for'])
    ->create();
```

Here is what happens to a field value:

```
"<span>Some people are worth melting for</span>"
   │
   ▼ Char filter: strip HTML
"Some people are worth melting for"
   │
   ▼ Tokenizer: whitespace
"Some" "people" "are" "worth" "melting" "for"
   │
   ▼ Token filter: lowercase
"some" "people" "are" "worth" "melting" "for"
   │
   ▼ Token filter: stopwords ("are", "for")
"some" "people" "worth" "melting"
```

Elasticsearch stores the four tokens at the bottom. It does not store the HTML, the capital letters, or the stopwords.

## Queries go through the same analyzer

At search time, the query string takes the same path:

```
"People Worth Melting"
   │
   ▼ no HTML to strip
   ▼ whitespace tokenizer
"People" "Worth" "Melting"
   │
   ▼ lowercase, stopwords
"people" "worth" "melting"
```

All three query tokens exist in the document, so the document matches. If only the document side were lowercased, `People` would never equal `people`, and the same search would find nothing.

## Filter order changes the result

Token filters run in the order you add them. The `stop` filter compares tokens case-sensitively, so a stopword list in lowercase only works after `lowercase()`:

```
lowercase(), then stopwords()        stopwords(), then lowercase()

"For" → "for" → (dropped)            "For" → "For" (kept) → "for"
```

Add normalizing filters such as `lowercase()` and `asciiFolding()` before the filters that compare against a word list.

## Where Sigmie attaches an analyzer

Sigmie builds analyzers at two levels:

```
newIndex('movies')
├─ analyzer "default"            built from the index builder methods
│                                 used by every text field without its own
└─ properties
   ├─ description  → "default"
   └─ email        → "email_field_analyzer"
                      built in withNewAnalyzer(), used only by this field
```

The analysis methods on the index builder, such as `lowercase()` or `tokenizeOnWhitespaces()`, build the analyzer named `default`. Elasticsearch uses `default` for every text field that has no analyzer of its own. With no analysis methods, `default` uses the word boundaries tokenizer and no filters.

`withNewAnalyzer()` on a text property builds an analyzer for that field only. Sigmie names it after the field path with a `_field_analyzer` suffix and uses it for both indexing and search on that field. Some property types, such as `title()`, `email()`, and `path()`, build their own field analyzer. See [Mappings](https://sigmie.com/docs/v2/mappings.md) for per-field analyzers.

`language()` swaps the index builder for one with language-specific filters. See [Languages](https://sigmie.com/docs/v2/language.md).

## Inspect the tokens an analyzer produces

`analyze()` sends text through one of the index's analyzers and returns the tokens. The second argument names the analyzer and defaults to `default`:

```php
$sigmie->index('quotes')->analyze('<b>Worth</b> melting for');
// ["worth", "melting"]

$sigmie->index('contacts')->analyze('nico@sigmie.com', 'email_field_analyzer');
```

Check the tokens before you reindex. A document token that differs from the query token never matches.


---

<!-- source: https://sigmie.com/docs/v2/tokenizers -->

# Tokenizers

The tokenizer splits a string into tokens. It is the middle stage of the [analysis pipeline](https://sigmie.com/docs/v2/analysis.md): it runs after the [char filters](https://sigmie.com/docs/v2/char-filters.md) and before the [token filters](https://sigmie.com/docs/v2/token-filters.md).

An analyzer has exactly one tokenizer. Each tokenizer method replaces the previous one, so the last call wins. Without a tokenizer method, an analyzer uses word boundaries.

The index builder and `NewAnalyzer` have the same tokenizer methods. The examples call them on `$analyzer`, the `NewAnalyzer` that `withNewAnalyzer()` passes to its callback.

| Builder method | Class | Elasticsearch type | Token boundaries |
|----------------|-------|--------------------|------------------|
| `tokenizeOnWordBoundaries()` | `WordBoundaries` | `standard` | Unicode word boundaries |
| `tokenizeOnWhitespaces()` | `Whitespace` | `whitespace` | Whitespace |
| `dontTokenize()` | `Noop` | `keyword` | None. The whole input is one token. |
| `tokenizeOnPattern($pattern, $flags)` | `Pattern` | `pattern` | Each match of a Java regex |
| `tokenizeOnSimplePattern($pattern)` | `SimplePatternSplit` | `simple_pattern_split` | Each match of a Lucene regex |
| `tokenizeOnPatternMatch($pattern)` | `SimplePattern` | `simple_pattern` | Each match becomes a token |
| `tokenizePathHierarchy($delimiter)` | `PathHierarchy` | `path_hierarchy` | Each path level |
| `tokenizeOnNonLetter()` | `NonLetter` | `letter` | Any character that is not a letter |
| None | `Ngram` | `ngram` | Overlapping character sequences |
| `tokenizer(Tokenizer $tokenizer)` | Any tokenizer instance | Set by the instance | Set by the instance |

Every builder method also takes an optional `$name` as its last argument. Without one, Sigmie generates a name from a prefix and a random suffix. The class constructors take the name as their first argument.

## Word boundaries

Splits on Unicode word boundaries and drops most punctuation. This is the default tokenizer.

```php
use Sigmie\Index\Analysis\Tokenizers\WordBoundaries;

$analyzer->tokenizeOnWordBoundaries();

// Or, with the class:
$analyzer->tokenizer(new WordBoundaries(name: 'word_boundaries', maxTokenLength: 255));
```

```
"Aw shucks, pluto. I can't be mad at ya!"

→ "Aw"
→ "shucks"
→ "pluto"
→ "I"
→ "can't"
→ "be"
→ "mad"
→ "at"
→ "ya"
```

`maxTokenLength` defaults to `255`. Elasticsearch splits a longer token into pieces of that length. Only the class accepts `maxTokenLength`. `tokenizeOnWordBoundaries()` always uses `255`.

## Whitespace

Splits on whitespace only. Punctuation stays attached to the word next to it.

```php
use Sigmie\Index\Analysis\Tokenizers\Whitespace;

$analyzer->tokenizeOnWhitespaces();

// Or, with the class:
$analyzer->tokenizer(new Whitespace(name: 'whitespace_tokenizer'));
```

```
"Aw shucks, pluto. I can't be mad at ya!"

→ "Aw" "shucks," "pluto." "I" "can't" "be" "mad" "at" "ya!"
```

## No-op

Keeps the whole input as one token. Use it for exact matches on a text field.

```php
use Sigmie\Index\Analysis\Tokenizers\Noop;

$analyzer->dontTokenize();

// Or, with the class:
$analyzer->tokenizer(new Noop(name: 'noop_tokenizer'));
```

```
"If you ain't scared, you ain't alive."

→ "If you ain't scared, you ain't alive."
```

## Pattern

Splits at each match of a Java regular expression. The matched text is not part of any token.

```php
use Sigmie\Index\Analysis\Tokenizers\Pattern;

$analyzer->tokenizeOnPattern(',');

// Or, with the class:
$analyzer->tokenizer(new Pattern(name: 'comma_tokenizer', pattern: ','));
```

```
"Though at times it may feel like the sky is falling around you, never give up, for every day is a new day"

→ "Though at times it may feel like the sky is falling around you"
→ " never give up"
→ " for every day is a new day"
```

The tokens keep their leading spaces. Add the [`trim()` token filter](https://sigmie.com/docs/v2/token-filters.md#trim) to remove them.

The optional `$flags` argument takes Java regex flags separated by `|`, for example `'CASE_INSENSITIVE|COMMENTS'`:

```php
$analyzer->tokenizeOnPattern('\s+and\s+', flags: 'CASE_INSENSITIVE');
```

## Simple pattern split

Splits at each match of a Lucene regular expression. Lucene regex has fewer features than Java regex, and Elasticsearch runs it faster.

```php
use Sigmie\Index\Analysis\Tokenizers\SimplePatternSplit;

$analyzer->tokenizeOnSimplePattern('_');

// Or, with the class:
$analyzer->tokenizer(new SimplePatternSplit(name: 'underscore_tokenizer', pattern: '_'));
```

```
"alpha_beta_gamma"

→ "alpha"
→ "beta"
→ "gamma"
```

## Simple pattern

Turns each match of a Lucene regular expression into a token and drops the rest of the text. It does the opposite of simple pattern split.

```php
use Sigmie\Index\Analysis\Tokenizers\SimplePattern;

$analyzer->tokenizeOnPatternMatch("'.*'");

// Or, with the class:
$analyzer->tokenizer(new SimplePattern(name: 'quoted_tokenizer', pattern: "'.*'"));
```

```
"I remember daddy told me 'Fairytales can come true'."

→ "'Fairytales can come true'"
```

> **Note:** The two method names look alike. `tokenizeOnSimplePattern()` splits at matches. `tokenizeOnPatternMatch()` keeps only the matches.

The `SimplePattern` class also takes a `$flags` argument. The `simple_pattern` tokenizer does not apply it: with `[A-Z]+` and `CASE_INSENSITIVE`, the input `abc ABC DEF 123` still produces only `ABC` and `DEF`.

## Path hierarchy

Produces one token for each level of a path.

```php
use Sigmie\Index\Analysis\Tokenizers\PathHierarchy;

$analyzer->tokenizePathHierarchy(delimiter: '/');

// Or, with the class:
$analyzer->tokenizer(new PathHierarchy(name: 'path_tokenizer', delimiter: '/'));
```

```
"Disney/Movies/Musical/Sleeping Beauty"

→ "Disney"
→ "Disney/Movies"
→ "Disney/Movies/Musical"
→ "Disney/Movies/Musical/Sleeping Beauty"
```

The delimiter defaults to `/`. Every path under `Disney/Movies` has the token `Disney/Movies`, so a term query for `Disney/Movies` matches all of them.

## Non-letter

Splits at every character that is not a letter. Digits and punctuation are dropped.

```php
use Sigmie\Index\Analysis\Tokenizers\NonLetter;

$analyzer->tokenizeOnNonLetter();

// Or, with the class:
$analyzer->tokenizer(new NonLetter(name: 'letter_tokenizer'));
```

```
"To infinity … and beyond!"

→ "To"
→ "infinity"
→ "and"
→ "beyond"
```

## N-gram

Produces every character sequence between `minGram` and `maxGram` characters long. There is no builder method. Pass an instance to `tokenizer()`.

```php
use Sigmie\Index\Analysis\Tokenizers\Ngram;

$analyzer->tokenizer(new Ngram(name: 'two_gram_tokenizer', minGram: 2, maxGram: 2));
```

```
"abcd"

→ "ab"
→ "bc"
→ "cd"
```

| Argument | Default | Elasticsearch setting |
|----------|---------|-----------------------|
| `minGram` | `3` | `min_gram` |
| `maxGram` | `3` | `max_gram` |
| `tokenChars` | `['letter']` | `token_chars` |

## See also

- [Text Analysis](https://sigmie.com/docs/v2/analysis.md): the full pipeline.
- [Token Filters](https://sigmie.com/docs/v2/token-filters.md): changes to tokens after the tokenizer.
- [Character Filters](https://sigmie.com/docs/v2/char-filters.md): changes to the string before the tokenizer.


---

<!-- source: https://sigmie.com/docs/v2/token-filters -->

# Token Filters

Token filters change, add, or remove tokens after the [tokenizer](https://sigmie.com/docs/v2/tokenizers.md) runs. An analyzer runs its token filters in the order you add them. For why the order matters, see [Filter order changes the result](https://sigmie.com/docs/v2/analysis.md#filter-order-changes-the-result).

The index builder and `NewAnalyzer` have the same token filter methods. The examples call them on `$analyzer`, the `NewAnalyzer` that `withNewAnalyzer()` passes to its callback. For English, German, and Greek filters, see [Languages](https://sigmie.com/docs/v2/language.md).

| Builder method | Elasticsearch type | Effect |
|----------------|--------------------|--------|
| `lowercase()` | `lowercase` | Lowercases each token |
| `uppercase()` | `uppercase` | Uppercases each token |
| `trim()` | `trim` | Removes leading and trailing whitespace |
| `asciiFolding()` | `asciifolding` | Replaces accented characters with ASCII |
| `decimalDigit()` | `decimal_digit` | Replaces Unicode digits with `0` to `9` |
| `stopwords(array $stopwords)` | `stop` | Removes listed tokens |
| `stemming(array $stemming)` | `stemmer_override` | Replaces listed words with a stem |
| `keywords(array $keywords)` | `keyword_marker` | Protects listed tokens from stemmers |
| `synonyms(array $synonyms, bool $expand = true)` | `synonym` | Adds or replaces synonyms |
| `oneWaySynonyms(array $synonyms)` | `synonym` | Replaces variants with one term |
| `twoWaySynonyms(array $synonyms)` | `synonym` | Makes terms equivalent |
| `unique(bool $onlyOnSamePosition = false)` | `unique` | Removes duplicate tokens |
| `tokenLimit(int $maxTokenCount)` | `limit` | Keeps the first N tokens |
| `truncate(int $length = 10)` | `truncate` | Cuts each token to a length |
| `shingle(int $minSize = 2, int $maxSize = 2)` | `shingle` | Adds word n-grams |
| `tokenFilter(TokenFilter $tokenFilter)` | Set by the instance | Adds any token filter instance |

## Token filter names

Each builder method takes an optional `$name` as its last argument. Without one, Sigmie generates a name from the method's prefix and a random suffix, such as `stopwords` followed by five random characters. A name must be unique in the index. Adding a token filter with a name that the index already has throws an `Exception` with the message ``Token filter `<name>` already exists.``

## Lowercase and uppercase

Change the case of each token.

```php
$analyzer->lowercase();
$analyzer->uppercase();
```

```
"You" "better" "be" "back" "ASAP"
   │
   ▼ Lowercase
"you" "better" "be" "back" "asap"
```

Without `lowercase()`, the token `Matrix` does not match a query for `matrix`.

## Trim

Removes leading and trailing whitespace from each token.

```php
$analyzer->trim();
```

```
" never give up"   →   "never give up"
" for every day"   →   "for every day"
```

The [pattern tokenizer](https://sigmie.com/docs/v2/tokenizers.md#pattern) can leave whitespace at the start of a token. Whitespace and word boundary tokens have none to remove.

## ASCII folding

Replaces letters with diacritics by their ASCII form.

```php
$analyzer->asciiFolding();
```

```
"manténgase"   →   "mantengase"
```

With `asciiFolding()` on the field, a query matches whether or not the user types the accent.

## Decimal digit

Replaces digits from other Unicode scripts with the ASCII digits `0` to `9`.

```php
$analyzer->decimalDigit();
```

```
"໑" "໒" "໓" "໔" "໕"     (Lao digits)
   │
   ▼ Decimal digit
"1" "2" "3" "4" "5"
```

## Stopwords

Removes each token that is in the list. The comparison is case-sensitive, so add `lowercase()` first when the list is in lowercase.

```php
$analyzer->stopwords(['but', 'not']);
```

```
"Ladies" "do" "not" "start" "fights" "but" "they" "can" "finish" "them"
   │
   ▼ Stopwords ("not", "but")
"Ladies" "do" "start" "fights" "they" "can" "finish" "them"
```

## Stemming

Replaces listed words with a stem that you choose. Each rule is a pair: the stem first, then the words that map to it.

```php
$analyzer->stemming([
    ['go', ['going']],
    ['mouse', ['mice']],
]);
```

```
"Where" "are" "you" "going"
   │
   ▼ Stemming
"Where" "are" "you" "go"
```

`stemming()` applies only the rules you list. For algorithmic stemmers, see [Languages](https://sigmie.com/docs/v2/language.md).

## Keywords

Marks tokens as keywords. Stemmers later in the chain leave keyword tokens unchanged. Add `keywords()` before the stemmer it guards.

```php
$analyzer
    ->keywords(['going'])
    ->stemming([
        ['go', ['going']],
    ]);
```

```
"Where" "are" "you" "going"
   │
   ▼ Keywords mark "going"
   ▼ Stemming skips "going"
"Where" "are" "you" "going"
```

## Synonyms

Sigmie has three synonym methods. All three create an Elasticsearch `synonym` filter.

### One-way synonyms

Replaces each variant with one term. Each rule is a pair: the term first, then its variants.

```php
$analyzer->oneWaySynonyms([
    ['ipod', ['i-pod', 'i pod']],
]);
```

Sigmie writes the rule `i-pod, i pod => ipod`. The filter turns `i-pod` and `i pod` into `ipod` and leaves `ipod` unchanged. The same analyzer runs on queries, so a search for any of the three spellings matches a document with any of the others.

### Two-way synonyms

Makes every term in a group equivalent. Each group is a list of terms.

```php
$analyzer->twoWaySynonyms([
    ['joy', 'fun'],
    ['friend', 'buddy', 'partner'],
]);
```

Sigmie writes one rule for each group, such as `joy, fun`, with `expand` set to `true`. Each term produces every term in its group at the same position:

```
"It's" "kind" "of" "fun" "to" "do" "the" "impossible"
   │
   ▼ Synonyms (joy, fun)
"It's" "kind" "of" "fun" "joy" "to" "do" "the" "impossible"
```

### The `synonyms()` method

Takes both rule formats in one list and sets `expand` yourself. `expand` defaults to `true`.

```php
$analyzer->synonyms([
    ['joy', 'fun'],
    ['ipod', ['i-pod', 'i pod']],
], expand: true);
```

With `expand: false`, a group such as `joy, fun` maps every term to the first term, `joy`. One-way rules ignore `expand`.

## Unique

Removes duplicate tokens.

```php
$analyzer->unique();
```

```
"I" "was" "hiding" "under" "your" "porch" "because" "I" "love" "you"
   │
   ▼ Unique
"I" "was" "hiding" "under" "your" "porch" "because" "love" "you"
```

With `onlyOnSamePosition: true`, the filter removes a duplicate only when it is at the same position, for example a synonym equal to the original token.

## Token limit

Keeps the first `maxTokenCount` tokens and drops the rest. `maxTokenCount` has no default.

```php
$analyzer->tokenLimit(maxTokenCount: 5);
```

```
"I" "was" "hiding" "under" "your" "porch" "because" "I" "love" "you"
   │
   ▼ Token limit 5
"I" "was" "hiding" "under" "your"
```

## Truncate

Cuts each token to at most `length` characters. `length` defaults to `10`.

```php
$analyzer->truncate(length: 10);
```

```
"Supercalifragilisticexpialidocious"
   │
   ▼ Truncate 10
"Supercalif"
```

## Shingle

Adds tokens made of neighboring words, from `minSize` to `maxSize` words long. Both default to `2`. The original single-word tokens stay.

```php
$analyzer->shingle(minSize: 2, maxSize: 2);
```

```
"quick" "brown" "fox"
   │
   ▼ Shingle 2
"quick" "quick brown" "brown" "brown fox" "fox"
```

## Custom token filters

`tokenFilter()` adds any object that implements `Sigmie\Index\Contracts\TokenFilter`. To use a filter from an Elasticsearch plugin, extend the `TokenFilter` class and return the plugin's type:

```php
use Sigmie\Index\Analysis\TokenFilter\TokenFilter;

use function Sigmie\Functions\name_configs;

class Greeklish extends TokenFilter
{
    public function type(): string
    {
        return 'skroutz_greeklish';
    }

    public static function fromRaw(array $raw): static
    {
        [$name] = name_configs($raw);

        return new static($name);
    }
}

$analyzer->tokenFilter(new Greeklish('greeklish'));
```

When Sigmie reads an existing index, for example with `$sigmie->index()`, it rebuilds each filter from the Elasticsearch `type`. A type that Sigmie does not know becomes a generic filter that keeps its settings. To get your own class back, register it by type:

```php
TokenFilter::filterMap([
    'skroutz_greeklish' => Greeklish::class,
]);
```

The keys are Elasticsearch filter types, not filter names. `CharFilter::filterMap()` and `Tokenizer::filterMap()` do the same for char filters and tokenizers. Those two have no generic fallback and throw an `Exception` for an unknown type.

## See also

- [Tokenizers](https://sigmie.com/docs/v2/tokenizers.md): the stage before token filters.
- [Character Filters](https://sigmie.com/docs/v2/char-filters.md): changes to the string before the tokenizer.
- [Text Analysis](https://sigmie.com/docs/v2/analysis.md): the full pipeline.
- [Languages](https://sigmie.com/docs/v2/language.md): stemmers, stopwords, and normalizers for English, German, and Greek.


---

<!-- source: https://sigmie.com/docs/v2/char-filters -->

# Character Filters

Character filters change the raw string before the [tokenizer](https://sigmie.com/docs/v2/tokenizers.md) splits it. An analyzer runs its char filters in the order you add them. For where char filters sit in the pipeline, see [Text Analysis](https://sigmie.com/docs/v2/analysis.md).

The index builder and `NewAnalyzer` have the same char filter methods. The examples call them on `$analyzer`, the `NewAnalyzer` that `withNewAnalyzer()` passes to its callback.

| Builder method | Class | Elasticsearch type |
|----------------|-------|--------------------|
| `stripHTML(?string $name = null)` | `HTMLStrip` | `html_strip` |
| `mapChars(array $mappings, ?string $name = null)` | `Mapping` | `mapping` |
| `patternReplace(string $pattern, string $replace, ?string $name = null)` | `Pattern` | `pattern_replace` |
| `charFilter(CharFilter $charFilter)` | Any char filter instance | Set by the instance |

## Char filter names

Each builder method takes an optional `$name`. Without one, Sigmie generates a name from a prefix and a random suffix, such as `html_strip` followed by five random characters. A name must be unique in the index. Adding a char filter with a name that the index already has throws an `Exception` with the message ``Char filter name `<name>` already exists.``

## HTML strip

Removes HTML tags and decodes HTML entities.

```php
use Sigmie\Index\Analysis\CharFilter\HTMLStrip;

$analyzer->stripHTML();

// Or, with the class:
$analyzer->charFilter(new HTMLStrip);
```

```
"<span>Some people are worth melting for.</span>"
   │
   ▼ Strip HTML
"Some people are worth melting for."
```

The `HTMLStrip` class uses the name `html_strip` when you pass no name.

## Character mapping

Replaces each key with its value. Keys are literal strings, not regular expressions.

```php
use Sigmie\Index\Analysis\CharFilter\Mapping;

$analyzer->mapChars([
    ':)' => 'happy',
    ':(' => 'sad',
]);

// Or, with the class:
$analyzer->charFilter(new Mapping(
    name: 'emoticons',
    mappings: [
        ':)' => 'happy',
        ':(' => 'sad',
    ],
));
```

```
"Even miracles take a little time. :)"
   │
   ▼ Map chars (":)" → "happy")
"Even miracles take a little time. happy"
```

## Pattern replace

Replaces every match of a Java regular expression. The replacement can refer to capture groups as `$1`, `$2`, and so on.

```php
use Sigmie\Index\Analysis\CharFilter\Pattern;

$analyzer->patternReplace(pattern: ':D|:\)', replace: 'happy');

// Or, with the class:
$analyzer->charFilter(new Pattern(
    name: 'smileys',
    pattern: ':D|:\)',
    replacement: 'happy',
));
```

```
"This is the perfect time to panic! :D :)"
   │
   ▼ Pattern replace (":D|:\)" → "happy")
"This is the perfect time to panic! happy happy"
```

> **Note:** The builder argument is `replace`. The `Pattern` constructor argument is `replacement`.

`mapChars()` matches fixed strings. `patternReplace()` matches a pattern, so it handles alternatives, variable-length matches, and anchors.

## See also

- [Tokenizers](https://sigmie.com/docs/v2/tokenizers.md): the stage after char filters.
- [Token Filters](https://sigmie.com/docs/v2/token-filters.md): changes to individual tokens.
- [Text Analysis](https://sigmie.com/docs/v2/analysis.md): the full pipeline.


---

<!-- source: https://sigmie.com/docs/v2/language -->

# Languages

Sigmie has index builders for English, German, and Greek. Each builder adds token filters for its language: stemmers, a stopword list, and a lowercase filter. The German builder also has a normalizer.

`language()` takes a language instance and returns that language's builder. The builder has every method of the index builder, plus the language filter methods:

```php
use Sigmie\Languages\English\English;

$sigmie->newIndex('articles')
    ->language(new English)
    ->properties($props)
    ->englishLowercase()
    ->englishStopwords()
    ->englishStemmer()
    ->create();
```

> **Warning:** Call `language()` directly after `newIndex()`. The language builder copies only the alias, the shard and replica counts, the registered APIs, and the collection hooks. It drops properties, token filters, char filters, and the tokenizer that you set before `language()`.

The language filters are token filters on the index `default` analyzer. They run in the order you add them, together with any generic [token filters](https://sigmie.com/docs/v2/token-filters.md). Add the lowercase filter before the stopwords and the stemmer, because both compare lowercase words.

Each method takes an optional `$name`. Without one, the filter uses the default name in the tables below. A name must be unique in the index.

## English

```php
use Sigmie\Languages\English\English;

$sigmie->newIndex('articles')
    ->language(new English)
    ->englishLowercase()
    ->englishPossessiveStemming()
    ->englishStopwords()
    ->englishStemmer()
    ->create();
```

| Method | Default name | Elasticsearch filter | Effect |
|--------|--------------|----------------------|--------|
| `englishLowercase()` | `english_lowercase` | `lowercase` | Lowercases each token |
| `englishStopwords()` | `english_stopwords` | `stop`, `stopwords: _english_` | Removes English stopwords |
| `englishPossessiveStemming()` | `english_stemmer_possessive` | `stemmer`, `language: possessive_english` | Removes a trailing `'s` |
| `englishStemmer()` | `english_stemmer` | `stemmer`, `language: english` | Porter stemmer |
| `englishPorter2Stemmer()` | `english_stemmer_porter_2` | `stemmer`, `language: porter2` | Snowball Porter2 stemmer |
| `englishLightStemmer()` | `english_stemmer_light` | `stemmer`, `language: light_english` | KStem stemmer |
| `englishMinimalStemmer()` | `english_stemmer_minimal` | `stemmer`, `language: minimal_english` | Removes plural endings |
| `englishLovinsStemmer()` | `english_stemmer_lovins` | `stemmer`, `language: lovins` | Lovins stemmer |

Use one stemmer per analyzer. A second stemmer stems tokens that the first one already changed. `englishPossessiveStemming()` is not a full stemmer, so you can add it before one.

With `englishStemmer()`, the text `running relational` produces the tokens `run` and `relat`.

> **Warning:** On Elasticsearch, an analyzer with `englishLovinsStemmer()` returns no tokens. Use another English stemmer there.

## German

```php
use Sigmie\Languages\German\German;

$sigmie->newIndex('artikel')
    ->language(new German)
    ->germanLowercase()
    ->germanStopwords()
    ->germanNormalize()
    ->germanStemmer()
    ->create();
```

| Method | Default name | Elasticsearch filter | Effect |
|--------|--------------|----------------------|--------|
| `germanLowercase()` | `german_lowercase` | `lowercase` | Lowercases each token |
| `germanStopwords()` | `german_stopwords` | `stop`, `stopwords: _german_` | Removes German stopwords |
| `germanNormalize()` | `german_normalization` | `german_normalization` | Folds umlauts and `ß` |
| `germanStemmer()` | `german_stemmer` | `stemmer`, `language: german` | German stemmer |
| `germanStemmer2()` | `german_stemmer_2` | `stemmer`, `language: german2` | German2 stemmer |
| `germanLightStemmer()` | `german_stemmer_light` | `stemmer`, `language: light_german` | Light German stemmer |
| `germanMinimalStemmer()` | `german_stemmer_minimal` | `stemmer`, `language: minimal_german` | Minimal German stemmer |

`germanNormalize()` replaces `ä`, `ö`, and `ü` with `a`, `o`, and `u`, and `ß` with `ss`. With it, `Tür` and `Tur` produce the same token, so a query matches whether or not the user types the umlaut. Add it after `germanStopwords()`, because the stopword list contains umlauts.

## Greek

```php
use Sigmie\Languages\Greek\Greek;

$sigmie->newIndex('arthra')
    ->language(new Greek)
    ->greekLowercase()
    ->greekStopwords()
    ->greekStemmer()
    ->create();
```

| Method | Default name | Elasticsearch filter | Effect |
|--------|--------------|----------------------|--------|
| `greekLowercase()` | `greek_lowercase` | `lowercase`, `language: greek` | Lowercases, removes accents, and changes `ς` to `σ` |
| `greekStopwords()` | `greek_stopwords` | `stop`, `stopwords: _greek_` | Removes Greek stopwords |
| `greekStemmer()` | `greek_stemmer` | `stemmer`, `language: greek` | Greek stemmer |

Use `greekLowercase()` instead of the generic `lowercase()` for Greek text. The Greek stemmer expects tokens without accents and with `σ` for every sigma, which only `greekLowercase()` produces. Add it before `greekStemmer()`.

## Language filters on one field

The language builders configure the index `default` analyzer. To give one field its own language, pass the filter classes to `tokenFilter()` in [`withNewAnalyzer()`](https://sigmie.com/docs/v2/mappings.md):

```php
use Sigmie\Index\NewAnalyzer;
use Sigmie\Languages\English\Filter\Stemmer as EnglishStemmer;
use Sigmie\Languages\German\Filter\Normalize as GermanNormalize;
use Sigmie\Languages\German\Filter\Stemmer as GermanStemmer;

$props->text('description_en')
    ->withNewAnalyzer(function (NewAnalyzer $analyzer) {
        $analyzer->lowercase();
        $analyzer->tokenFilter(new EnglishStemmer('description_en_stemmer'));
    });

$props->text('description_de')
    ->withNewAnalyzer(function (NewAnalyzer $analyzer) {
        $analyzer->lowercase();
        $analyzer->tokenFilter(new GermanNormalize('description_de_normalize'));
        $analyzer->tokenFilter(new GermanStemmer('description_de_stemmer'));
    });
```

Each filter class lives in `Sigmie\Languages\<Language>\Filter` and takes its name as the only constructor argument. Pass a name to each instance. A class without one uses its default name, and a second filter with that name in the same index throws an `Exception`.

## See also

- [Text Analysis](https://sigmie.com/docs/v2/analysis.md): how analyzers turn text into tokens.
- [Token Filters](https://sigmie.com/docs/v2/token-filters.md): generic filters that work in any language.
- [Indices](https://sigmie.com/docs/v2/index.md): index configuration.


---

<!-- source: https://sigmie.com/docs/v2/filter-parser -->

# Filter Parser

The filter parser turns a filter string such as `active:true AND price:100..500` into an Elasticsearch `bool` query. It checks every field name against your properties.

Pass a filter string to `filters()` on `newSearch()`, or to `parse()` on `newQuery()`:

```php
$sigmie->newSearch('products')
    ->properties($props)
    ->filters('active:true AND stock>0 AND price:100..500')
    ->queryString('laptop')
    ->get();
```

To build the query without a search, create a `FilterParser`:

```php
use Sigmie\Mappings\NewProperties;
use Sigmie\Parse\FilterParser;

$props = new NewProperties;
$props->keyword('category');
$props->bool('active');
$props->number('stock');

$parser = new FilterParser($props());
$query = $parser->parse('active:true AND category:"sports"');
```

`parse()` returns a `Boolean` query. An empty string returns a query that matches every document.

## Exact values

```
category:"sports"
color:'red'
name:'John Doe'
status:active
```

Single and double quotes are interchangeable. A value without spaces or special characters needs no quotes.

### Numbers

```
price:100
stock:50
```

### Booleans

```
active:true
published:false
```

`true` and `false` are lowercase and have no quotes.

### Empty strings

```
database:""
```

`field:""` matches documents where the field holds an empty string.

## Field exists

```
email:*               # has any value
NOT email:*           # has no value
```

## Multiple values

A list matches documents with any of its values:

```
category:["sports", "action", "horror"]
status:["active", "pending"]
```

The parser trims spaces around each value. An empty list, `tags:[]`, matches no documents.

## Wildcards

```
phone:'*650'         # ends with 650
phone:'2353*'        # starts with 2353
title:'*manager*'    # contains "manager"
```

To match a literal `*`, escape it: `category:'A\*B'` matches the value `A*B`.

## Document IDs

```
_id:"dkKwMe4UBAUb2dMteRe2"
_id:["dkKwMe4UBAUb2dMteRe2", "wd6Me4UBAUb2dMJT"]
```

`_id` needs no entry in the properties.

## Ranges

### Comparison operators

```
price>=100
price<=200
stock>0
created_at>="2023-05-01"
```

The operators are `>`, `<`, `>=`, and `<=`.

### Inclusive range

```
price:100..500
created_at:"2023-01-01".."2023-12-31"
```

`price:100..500` is the same as `price>=100 AND price<=500`.

## Logical operators

```
active:true AND category:"sports" AND stock>0
category:"action" OR category:"horror"
NOT category:"sports"
active:true AND NOT stock:0
```

`AND`, `OR`, and `NOT` are case-insensitive. `AND` binds tighter than `OR`, so `a OR b AND c` means `a OR (b AND c)`. Use parentheses to group clauses:

```
active:true AND (category:"action" OR category:"horror") AND stock>0
```

> **Note:** Two clauses without an operator between them throw a `ParseException`:
>
> ```
> color:'red' size:'large'             # error
> color:'red' AND size:'large'         # OK
> ```

## Object fields

For fields inside an `object()` property, use dot notation:

```
contact.active:true
contact.name:"John Doe"
user.profile.settings.notifications:true
```

```php
$props->object('contact', function (NewProperties $p) {
    $p->bool('active');
    $p->name('name');
    $p->email('email');
});

$parser->parse('contact.active:true AND contact.name:"Alice"');
```

## Nested field filtering

For a `nested()` property, put the conditions in curly braces. All conditions inside the braces must match the same item of the list:

```
contact:{ active:true }
contact:{ name:"John Doe" AND verified:true }
```

```php
$props->nested('vehicles', function (NewProperties $p) {
    $p->keyword('make');
    $p->keyword('model');
});

$parser->parse("vehicles:{ make:'Powell Motors' AND model:'Canyonero' }");
```

This filter finds documents with one vehicle whose `make` is "Powell Motors" and whose `model` is "Canyonero". A document with a "Powell Motors" vehicle and a different "Canyonero" vehicle does not match.

Nested fields can hold nested fields:

```
contact:{ address:{ city:"Berlin" AND marker:"X" } }
```

The mapping decides the syntax:

| Mapping | Syntax |
|---------|--------|
| `object()` | `contact.active:true AND contact.city:"Berlin"` |
| `nested()` | `contact:{ active:true AND city:"Berlin" }` |

To return the nested items that matched the filter, see [inner hits](https://sigmie.com/docs/v2/search.md#matching-nested-items-inner-hits).

## Geo-location filtering

A `geoPoint()` field matches documents within a distance of a point:

```
location:1km[51.49,13.77]
location:5mi[40.7128,-74.0060]
contact:{ location:1km[51.16,13.49] AND active:true }
```

The format is `field:distance[lat,lon]`. The distance is a number followed by a unit: `km`, `m`, `cm`, `mm`, `mi`, `yd`, `ft`, `in`, or `nmi`.

> **Note:** A distance of zero matches no documents, even on the exact coordinates. Use a small positive distance:
>
> ```
> location:0km[51.16,13.49]       # matches nothing
> location:1m[51.16,13.49]        # OK
> ```

## Quotes and special characters

Escape a quote inside a value with a backslash. A single quote inside double quotes needs no escape:

```
description:"She said \"Hello World\""
title:'It\'s working'
team:"O'Brien"
```

Quote values that contain spaces, dashes, or other special characters:

```
status:'in-progress'
category:"crime & drama"
job_title:"Chief Executive Officer (CEO)"
industry:["Renewables & Environment"]
```

## Field rules

The parser checks each field against the properties:

| Field | Rule |
|-------|------|
| Not in the properties | Error: `Field <name> does not exist.` |
| `text` without `keyword()` | Error: `Field <name> is not filterable.` |
| `text` with `keyword()` | Filters on the `.keyword` sub-field. |
| `field:{...}` on a field that is not `nested()` | Error: `Field '<name>' isn't a nested field.` |

## Errors

The parser reports errors in two ways.

Syntax errors always throw a `ParseException`:

- Two clauses without an operator between them.
- Unbalanced parentheses, brackets, braces, or quotes.
- More than 32 levels of parentheses and `NOT`. `FilterParser::$maxNestingLevel` sets the limit.

Field errors, and a clause the parser cannot read, follow the `throwOnError` setting. When it is `true`, they throw a `ParseException`. When it is `false`, the parser replaces the clause with a clause that matches nothing and adds the error to `errors()`:

```php
use Sigmie\Parse\FilterParser;

$parser = new FilterParser($props(), throwOnError: false);

$parser->parse('subject_service:{ id:"23" }');

$parser->errors();   // [['message' => "Field 'subject_service' isn't a nested field."]]
```

The default depends on the caller:

| Caller | Default |
|--------|---------|
| `new FilterParser($props)` | Throws |
| `newQuery()->parse()` | Throws |
| `newSearch()->filters()` | Collects. The errors appear under the response's `errors` key. |

To make `filters()` throw, pass `throwOnError: true`:

```php
$sigmie->newSearch('products')
    ->properties($props)
    ->filters('nonexistent:"x"', throwOnError: true);   // throws ParseException
```

A collected error still changes the results. If a broader or empty result set is unsafe, pass `throwOnError: true`, or treat a non-empty `errors` array as a failed request.

The [Laravel AI tools](https://sigmie.com/docs/v2/laravel-ai.md#errors) always parse with `throwOnError: true`, so an agent sees the error instead of an empty result.

## Syntax summary

| Operation | Syntax | Example |
|-----------|--------|---------|
| Exact value | `field:"value"` | `category:"sports"` |
| Number | `field:123` | `price:100` |
| Boolean | `field:true` | `active:true` |
| Field exists | `field:*` | `email:*` |
| Any of | `field:[v1,v2]` | `status:["active","pending"]` |
| Wildcard | `field:'*pat*'` | `phone:'*650'` |
| Document IDs | `_id:[id1,id2]` | `_id:["a1","b2"]` |
| Range | `field:min..max` | `price:100..500` |
| Greater than | `field>value` | `stock>0` |
| Less than | `field<value` | `price<100` |
| AND | `a AND b` | `active:true AND stock>0` |
| OR | `a OR b` | `cat:"a" OR cat:"b"` |
| NOT | `NOT a` | `NOT category:"books"` |
| Object | `obj.field:value` | `contact.active:true` |
| Nested | `field:{condition}` | `contact:{active:true}` |
| Geo distance | `field:dist[lat,lon]` | `location:1km[51,13]` |


---

<!-- source: https://sigmie.com/docs/v2/sort-parser -->

# Sort Parser

The sort parser turns a sort string such as `_score rating:desc name:asc` into an Elasticsearch `sort` array. It checks every field name against your properties.

Pass a sort string to `sort()` on `newSearch()`, or to `sortString()` on `newQuery()`:

```php
$sigmie->newSearch('movies')
    ->properties($props)
    ->sort('_score rating:desc name:asc');

$sigmie->newQuery('movies')
    ->properties($props)
    ->sortString('_score rating:desc name:asc');
```

Each call replaces the previous sort. For the search builders, see [Sort](https://sigmie.com/docs/v2/search.md#sort) and [Sorting](https://sigmie.com/docs/v2/query.md#sorting).

To build the array without a search, create a `SortParser`:

```php
use Sigmie\Mappings\NewProperties;
use Sigmie\Parse\SortParser;

$props = new NewProperties;
$props->number('rating');
$props->text('name')->makeSortable();

$parser = new SortParser($props());
$sort = $parser->parse('_score rating:desc name:asc');
```

`$sort` holds:

```json
[
    "_score",
    { "rating": "desc" },
    { "name.sortable": "asc" }
]
```

## Syntax

```
_score:desc rating:desc name:asc
```

One space separates the clauses. Each clause is one of these forms:

| Clause | Sorts by |
|--------|----------|
| `_score` or `_score:desc` | Relevance, highest first. |
| `_doc` | Index order. |
| `field` | The field, ascending. |
| `field:asc` | The field, ascending. |
| `field:desc` | The field, descending. |
| `field[lat,lon]:unit:direction` | Distance from a point. See [Geo distance](#geo-distance). |

An empty string sorts by `_score`.

> **Note:** `_score:asc` is not allowed, because Elasticsearch cannot sort relevance ascending. Use `_score` or `_score:desc`.

## Field types

The parser picks the sort field from the property type:

| Property | Sort field |
|----------|------------|
| `keyword()` | The field itself. |
| `text()` with `makeSortable()` | The `.sortable` sub-field. |
| `text()` without `makeSortable()` | Error: `Field <name> is not sortable.` |
| `name()`, `title()`, `category()`, `email()`, `address()`, `searchableNumber()` | The `.sortable` sub-field. These types add it themselves. |
| `id()` | `id.sortable`. |
| Other types, such as `number()`, `date()`, and `bool()` | The field itself. |

A field inside a `nested()` property sorts with the nested path:

```json
{ "contact.name.sortable": { "nested": { "path": "contact" }, "order": "asc" } }
```

## Geo distance

A `geoPoint()` field sorts by its distance from a point:

```
location[40.71,-74.00]:km:asc
```

The format is `field[lat,lon]:unit:direction`. All four parts are required.

| Part | Values |
|------|--------|
| `lat` | -90 to 90 |
| `lon` | -180 to 180 |
| `unit` | `km`, `m`, `cm`, `mm`, `mi`, `yd`, `ft`, `in`, `nmi` |
| `direction` | `asc`, `desc` |

## Errors

The parser reports an error for:

- A field that is not in the properties.
- A `text` field without `makeSortable()`.
- `_score:asc`, or any direction on `_score` other than `desc`.
- A direction written after a space, such as `name asc`. Write `name:asc`.
- A geo clause on a field that is not a `geoPoint()`, or with an invalid unit, direction, or coordinate.

`new SortParser($props)` throws a `ParseException` on the first error. With `new SortParser($props, throwOnError: false)`, the parser skips the invalid clause and adds the error to `errors()`.

| Caller | Default |
|--------|---------|
| `new SortParser($props)` | Throws |
| `newSearch()->sort()` | Throws. Pass `throwOnError: false` to skip invalid clauses. |
| `newQuery()->sortString()` | Throws |


---

<!-- source: https://sigmie.com/docs/v2/connection -->

# Connection Setup

This page shows how to connect Sigmie to a production cluster: authentication, TLS, several nodes, and hosted services. For a first local connection, see [Installation](https://sigmie.com/docs/v2/installation.md).

Every connection goes through `Sigmie::create()`. It takes the hosts, the engine type, and a `config` array that Sigmie passes to the Guzzle HTTP client:

```php
use Sigmie\Sigmie;

$sigmie = Sigmie::create(
    hosts: ['https://elasticsearch.example.com:9200'],
    config: ['auth' => ['elastic', 'your-password']],
);
```

## Authenticate with a username and password

Pass the credentials in `auth`:

```php
use Sigmie\Sigmie;

$sigmie = Sigmie::create(
    hosts: ['https://elasticsearch.example.com:9200'],
    config: [
        'auth' => ['elastic', 'your-password'],
    ]
);
```

To build the client yourself, use `JSONClient::createWithBasic()` and wrap it in a connection:

```php
use Sigmie\Http\JSONClient;
use Sigmie\Base\Http\ElasticsearchConnection;
use Sigmie\Base\Drivers\Elasticsearch;
use Sigmie\Sigmie;

$client = JSONClient::createWithBasic(
    hosts: ['https://elasticsearch.example.com:9200'],
    username: 'elastic',
    password: 'your-password'
);

$connection = new ElasticsearchConnection($client, new Elasticsearch);
$sigmie = new Sigmie($connection);
```

## Authenticate with an API key

Create the key in Elasticsearch:

```bash
curl -X POST "https://elasticsearch.example.com:9200/_security/api_key" \
  -H 'Content-Type: application/json' \
  -u elastic:your-password \
  -d '{"name": "my-api-key", "expiration": "90d"}'
```

The response contains `id`, `api_key`, and `encoded`. The `encoded` value is the Base64 of `id:api_key`. Send it in the `Authorization` header:

```php
$sigmie = Sigmie::create(
    hosts: ['https://elasticsearch.example.com:9200'],
    config: [
        'headers' => [
            'Authorization' => 'ApiKey ' . base64_encode('id:api_key'),
        ],
    ]
);
```

## Authenticate with a bearer token

```php
$sigmie = Sigmie::create(
    hosts: ['https://elasticsearch.example.com:9200'],
    config: [
        'headers' => [
            'Authorization' => 'Bearer your-token-here',
        ],
    ]
);
```

## Trust a custom CA certificate

To verify the cluster against your own certificate authority, set `verify` to the path of the CA file:

```php
$sigmie = Sigmie::create(
    hosts: ['https://elasticsearch.example.com:9200'],
    config: ['verify' => '/path/to/ca-certificate.pem'],
);
```

For a local cluster with a self-signed certificate, turn verification off:

```php
$sigmie = Sigmie::create(
    hosts: ['https://localhost:9200'],
    config: ['verify' => false],
);
```

> **Warning:** Never turn off certificate verification in production.

## Connect to several nodes

List every node in `hosts`:

```php
$sigmie = Sigmie::create(
    hosts: [
        '10.0.0.1:9200',
        '10.0.0.2:9200',
        '10.0.0.3:9200',
    ],
    config: ['auth' => ['elastic', 'your-password']],
);
```

`hosts` also accepts one comma-separated string, such as `'10.0.0.1:9200,10.0.0.2:9200'`.

The transport sends requests to the nodes in round-robin order. When a node fails with a network error, the transport marks it dead and later requests skip it. Sigmie does not retry the request that failed.

## Connect to OpenSearch

Set `engine` to `SearchEngineType::OpenSearch`:

```php
use Sigmie\Enums\SearchEngineType;

$sigmie = Sigmie::create(
    hosts: ['https://localhost:9200'],
    engine: SearchEngineType::OpenSearch,
    config: [
        'auth' => ['admin', 'MyStrongPass123!@#'],
        'verify' => false,
    ]
);
```

See [OpenSearch](https://sigmie.com/docs/v2/opensearch.md) for what changes on that engine.

## Connect to Elastic Cloud

With a username and password, use the deployment's Elasticsearch endpoint. Find it in the deployment dashboard under **Elasticsearch endpoint**:

```php
$sigmie = Sigmie::create(
    hosts: ['https://my-deployment.es.us-east-1.aws.found.io:9243'],
    config: ['auth' => ['elastic', 'your-cloud-password']],
);
```

With a Cloud ID and an API key, use `createForCloud()`:

```php
$sigmie = Sigmie::createForCloud(
    cloudId: 'my-deployment:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyQ...',
    apiKey: 'your-encoded-api-key',
);
```

For an Elasticsearch Serverless project, use `createForServerless()` with the project endpoint and an API key:

```php
$sigmie = Sigmie::createForServerless(
    host: 'https://my-project.es.us-east-1.aws.elastic.cloud',
    apiKey: 'your-encoded-api-key',
);
```

## Connect to AWS OpenSearch Service with IAM

AWS OpenSearch Service can require requests signed with AWS Signature Version 4. Install `aws/aws-sdk-php`, sign each request in a Guzzle handler, and pass the handler in `config`:

```php
use Aws\Credentials\CredentialProvider;
use Aws\Signature\SignatureV4;
use GuzzleHttp\HandlerStack;
use GuzzleHttp\Middleware;
use Sigmie\Http\JSONClient;
use Sigmie\Base\Http\ElasticsearchConnection;
use Sigmie\Base\Drivers\Opensearch;
use Sigmie\Sigmie;

$credentials = CredentialProvider::defaultProvider()()->wait();
$signer = new SignatureV4('es', 'us-east-1');
$handler = HandlerStack::create();
$handler->push(Middleware::mapRequest(fn ($request) =>
    $signer->signRequest($request, $credentials)
));

$client = JSONClient::create(
    hosts: ['https://search-domain.us-east-1.es.amazonaws.com:443'],
    config: ['handler' => $handler]
);

$connection = new ElasticsearchConnection($client, new Opensearch);
$sigmie = new Sigmie($connection);
```

For a domain with username and password access, see [OpenSearch](https://sigmie.com/docs/v2/opensearch.md).

## Set timeouts

Raise `timeout` for long bulk writes:

```php
$sigmie = Sigmie::create(
    hosts: ['127.0.0.1:9200'],
    config: [
        'connect_timeout' => 15,
        'timeout' => 120,
    ]
);
```

## Read the connection from the environment

```php
$sigmie = Sigmie::create(
    hosts: $_ENV['ELASTICSEARCH_HOSTS'],
    config: [
        'auth' => [$_ENV['ES_USER'], $_ENV['ES_PASSWORD']],
        'verify' => $_ENV['ES_VERIFY_SSL'] === 'true',
    ]
);
```

## Configuration options

The `config` array accepts any [Guzzle request option](https://docs.guzzlephp.org/en/stable/request-options.html). Your values replace Sigmie's defaults. These options matter most:

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `auth` | array | none | Basic auth credentials, `['username', 'password']`. |
| `headers` | array | none | Extra HTTP headers, such as `Authorization`. |
| `verify` | bool\|string | `true` | Verify the TLS certificate. A string is the path to a CA file. |
| `connect_timeout` | int\|float | `15` | Seconds to wait for a connection. Sigmie sets this default. |
| `timeout` | int\|float | `0` | Seconds to wait for the response. `0` waits without a limit. |
| `allow_redirects` | bool | `false` | Follow HTTP redirects. Sigmie sets this default. |
| `http_errors` | bool | `false` | Let Guzzle throw on 4xx and 5xx. Leave it off. Sigmie reads the status and throws its own `ElasticsearchException`. |
| `handler` | callable | Guzzle default | A Guzzle handler stack, used for request signing. |

## Verify the connection

`isConnected()` sends a request to the cluster root and returns `false` on any failure:

```php
if ($sigmie->isConnected()) {
    echo "Connected.\n";
}

foreach ($sigmie->indices() as $index) {
    echo $index->name . "\n";
}
```

## Fix common connection errors

**`cURL error 7: Failed to connect`.** The cluster is not running, or the host or port is wrong. Run `curl http://localhost:9200` to check.

**`cURL error 60: SSL certificate problem`.** The client does not trust the certificate. Set `verify` to the CA file path. In local development only, set `verify` to `false`.

**`401 Unauthorized`.** The cluster rejected the credentials. Check the username, password, or key, and check the cluster's security logs.

**`cURL error 28: Operation timed out`.** Raise `connect_timeout` and `timeout`. Bulk writes often need 60 to 120 seconds.


---

<!-- source: https://sigmie.com/docs/v2/docker -->

# Docker

This page shows how to run a search engine and local AI services with Docker Compose. The [Infinity](https://github.com/michaelfeil/infinity) services give you embeddings and reranking in development without a paid API.

The `docker-compose.yml` file is in the root of the [Sigmie repository](https://github.com/sigmie/sigmie). The Composer package does not include it. Clone the repository, or copy the file into your project. The `llm` service also needs `scripts/ollama-entrypoint.sh` from the repository.

## Services in the Compose file

| Service | Port | Model | Purpose |
|---------|------|-------|---------|
| `elasticsearch` | 9200 | none | Elasticsearch 9.1.3, single node, security off |
| `opensearch` | 9200 | none | OpenSearch 3, single node, user `admin`, password `MyStrongPass123!@#` |
| `embeddings` | 7997 | `BAAI/bge-small-en-v1.5` (384 dimensions) | Text embeddings |
| `reranker` | 7998 | `cross-encoder/ms-marco-MiniLM-L-6-v2` | Reranking |
| `image-embeddings` | 7996 | `wkcn/TinyCLIP-ViT-8M-16-Text-3M-YFCC15M` | Image and text embeddings |
| `llm` | 7999 | Ollama, `tinyllama` by default | Text generation for your application. Sigmie has no LLM client. |

> **Note:** Elasticsearch and OpenSearch both bind port 9200. Start one of them, not both.

## Start the services

Start only the services you need:

```bash
# Keyword search
docker compose up -d elasticsearch

# Semantic search
docker compose up -d elasticsearch embeddings

# Semantic search with reranking
docker compose up -d elasticsearch embeddings reranker

# Image search
docker compose up -d elasticsearch image-embeddings
```

The first start downloads the models, which takes several minutes. Later starts reuse the downloads.

To use OpenSearch, replace `elasticsearch` with `opensearch` in these commands.

## Check that the services are up

```bash
docker compose ps
curl http://localhost:7997/health                  # embeddings
curl http://localhost:7998/health                  # reranker
curl http://localhost:7996/health                  # image embeddings
curl http://localhost:9200/_cluster/health         # Elasticsearch
curl -k -u 'admin:MyStrongPass123!@#' \
    https://localhost:9200/_cluster/health         # OpenSearch
```

Each health endpoint returns HTTP 200 when the service is ready. Keep the OpenSearch credentials in single quotes, or the shell expands the `!` in the password.

## Connect Sigmie to the engine

For Elasticsearch:

```php
use Sigmie\Sigmie;

$sigmie = Sigmie::create(hosts: ['127.0.0.1:9200']);
```

For OpenSearch:

```php
use Sigmie\Sigmie;
use Sigmie\Enums\SearchEngineType;

$sigmie = Sigmie::create(
    hosts: ['https://localhost:9200'],
    engine: SearchEngineType::OpenSearch,
    config: [
        'auth' => ['admin', 'MyStrongPass123!@#'],
        'verify' => false,
    ]
);
```

## Register the local AI services

Register each service under a name. Mappings and searches refer to that name:

```php
use Sigmie\AI\APIs\InfinityClipApi;
use Sigmie\AI\APIs\InfinityEmbeddingsApi;
use Sigmie\AI\APIs\InfinityRerankApi;

$sigmie->registerApi('embeddings', new InfinityEmbeddingsApi(
    baseUrl: 'http://localhost:7997',
    model: 'BAAI/bge-small-en-v1.5',
));

$sigmie->registerApi('rerank', new InfinityRerankApi(
    baseUrl: 'http://localhost:7998',
    model: 'cross-encoder/ms-marco-MiniLM-L-6-v2',
));

$sigmie->registerApi('clip', new InfinityClipApi(
    baseUrl: 'http://localhost:7996',
    model: 'wkcn/TinyCLIP-ViT-8M-16-Text-3M-YFCC15M',
));
```

These URLs and models are also the constructor defaults, so `new InfinityEmbeddingsApi()` gives the same client.

To use the services, see [Semantic Search](https://sigmie.com/docs/v2/semantic-search.md) for embeddings and reranking, and [Retrieval and Agents](https://sigmie.com/docs/v2/rag.md) for retrieval with generation.

## Set service URLs and API keys

Copy `.env.example` to `.env` to change the service URLs:

```ini
LOCAL_EMBEDDING_URL=http://localhost:7997
LOCAL_RERANK_URL=http://localhost:7998
LOCAL_CLIP_URL=http://localhost:7996
```

Sigmie's test base class, `Sigmie\Testing\TestCase`, reads these variables. The library does not. Your application reads its own settings and passes them to `registerApi()`.

The same file holds keys for hosted providers:

```ini
OPENAI_API_KEY=sk-...
VOYAGE_API_KEY=pa-...
COHERE_API_KEY=...
MIXEDBREAD_API_KEY=...
```

## Read the logs

```bash
docker compose logs -f                # all services, follow
docker compose logs embeddings        # one service
```

## Reset the data

The services store their data in `./data/`:

```
./data/
├── embeddings/         # downloaded model
├── reranker/           # downloaded model
├── image-embeddings/   # downloaded model
├── llm/                # Ollama models
├── elasticsearch/      # indices and documents
└── opensearch/         # indices and documents
```

To start from nothing, stop the services and delete the directory:

> **Warning:** This deletes all indices, documents, and downloaded models.

```bash
docker compose down
rm -rf ./data/
```

## Plan memory and disk

These figures are approximate:

| Service | RAM | Disk |
|---------|-----|------|
| Embeddings | 1–2 GB | ~500 MB |
| Reranker | 1–2 GB | ~400 MB |
| Image embeddings | 1–2 GB | ~300 MB |
| Elasticsearch | 2–4 GB | depends on your data |
| OpenSearch | 2–4 GB | depends on your data |

For Elasticsearch with embeddings and reranking, give Docker at least 8 GB of memory.

## Fix common problems

**Port 9200 is already in use.** The other engine is running. Stop it, then start the one you want:

```bash
docker compose stop elasticsearch
docker compose up -d opensearch
```

**The embeddings service does not respond.** On the first start it is still downloading the model. Follow the logs and wait until `curl http://localhost:7997/health` returns HTTP 200:

```bash
docker compose logs -f embeddings
```

**A container stops with an out-of-memory error.** Give Docker more memory in the Docker Desktop settings.


---

<!-- source: https://sigmie.com/docs/v2/opensearch -->

# OpenSearch

Sigmie runs on OpenSearch with the same API it uses for Elasticsearch. You set the engine type when you connect. Index, document, and search code stays the same. Sigmie's test suite runs against OpenSearch 3.2.0.

## Connect to OpenSearch

Pass `SearchEngineType::OpenSearch` as the `engine`:

```php
use Sigmie\Sigmie;
use Sigmie\Enums\SearchEngineType;

$sigmie = Sigmie::create(
    hosts: ['https://localhost:9200'],
    engine: SearchEngineType::OpenSearch,
    config: [
        'auth' => ['admin', 'MyStrongPass123!@#'],
        'verify' => false,    // self-signed certificate in development
    ]
);
```

For TLS, API keys, and other connection options, see [Connection Setup](https://sigmie.com/docs/v2/connection.md).

## Move from Elasticsearch

Add the `engine` argument to `Sigmie::create()`:

```php
use Sigmie\Enums\SearchEngineType;

// Before
$sigmie = Sigmie::create(hosts: ['https://localhost:9200']);

// After
$sigmie = Sigmie::create(
    hosts: ['https://localhost:9200'],
    engine: SearchEngineType::OpenSearch,
);
```

The rest of your code does not change. Indices do not move between engines. Build each index again on the new cluster, then index your documents into it.

## Connect to AWS OpenSearch Service

For a domain with username and password access:

```php
$sigmie = Sigmie::create(
    hosts: ['https://your-domain.region.es.amazonaws.com'],
    engine: SearchEngineType::OpenSearch,
    config: ['auth' => ['username', 'password']],
);
```

For a domain that requires IAM-signed requests, see the AWS section of [Connection Setup](https://sigmie.com/docs/v2/connection.md).

## Semantic fields on OpenSearch

Define semantic fields the same way you do for Elasticsearch:

```php
use Sigmie\Mappings\NewProperties;

$props = new NewProperties;
$props->text('title')->semantic(api: 'embeddings', dimensions: 384);
$props->text('content')->semantic(api: 'embeddings', accuracy: 3, dimensions: 384);

$sigmie->newIndex('articles')->properties($props)->create();
```

Search them the same way too:

```php
$results = $sigmie->newSearch('articles')
    ->properties($props)
    ->semantic()
    ->queryString('quantum computing')
    ->size(20)
    ->get();
```

For accuracy levels, similarity, and providers, see [Semantic Search](https://sigmie.com/docs/v2/semantic-search.md).

## What Sigmie changes on OpenSearch

The engine type changes how Sigmie writes the mapping and the settings:

| Setting | Elasticsearch | OpenSearch |
|---------|---------------|------------|
| Vector field type | `dense_vector` | `knn_vector`, `lucene` engine, `hnsw` method |
| Index setting | none | `index.knn: true` |
| Vector index types | `hnsw`, `int8_hnsw`, `int4_hnsw`, `bbq_hnsw`, `flat`, `int8_flat`, `int4_flat`, `bbq_flat` | `hnsw` only |

Sigmie maps each `VectorSimilarity` case to the OpenSearch space type:

| `VectorSimilarity` | OpenSearch `space_type` |
|--------------------|-------------------------|
| `Cosine` (default) | `cosinesimil` |
| `DotProduct` | `innerproduct` |
| `MaxInnerProduct` | `innerproduct` |
| `Euclidean` | `l2` |

> **Warning:** A semantic field with an `indexType()` other than `hnsw` throws an exception on OpenSearch when Sigmie builds the mapping.


---

<!-- source: https://sigmie.com/docs/v2/laravel-scout -->

# Laravel Scout

`sigmie/elasticsearch-scout` is a [Laravel Scout](https://laravel.com/docs/scout) driver. Scout sends model writes and deletes to Elasticsearch, and `Model::search()` runs through Sigmie's [search builder](https://sigmie.com/docs/v2/search.md).

The driver connects with the Elasticsearch driver only. It has no OpenSearch option.

## Install the driver

1. Install Scout:

   ```bash
   composer require laravel/scout
   ```

2. Publish the Scout config:

   ```bash
   php artisan vendor:publish --provider="Laravel\Scout\ScoutServiceProvider"
   ```

3. Install the Sigmie driver:

   ```bash
   composer require sigmie/elasticsearch-scout
   ```

4. Set the Scout driver in `.env`:

   ```ini
   SCOUT_DRIVER=elasticsearch
   ```

5. To change the connection or the index settings, publish the driver config:

   ```bash
   php artisan vendor:publish --provider="Sigmie\ElasticsearchScout\ElasticsearchScoutServiceProvider"
   ```

   This creates `config/elasticsearch-scout.php`. See [Configuration reference](#configuration-reference) for its keys.

## Make a model searchable

Use the `Searchable` trait from `Sigmie\ElasticsearchScout`, not the one from Laravel Scout. The Sigmie trait includes Scout's trait and adds the methods the driver calls.

Define the index fields in `elasticsearchProperties()`. The trait declares it abstract, so every model must implement it:

```php
use Illuminate\Database\Eloquent\Model;
use Sigmie\ElasticsearchScout\Searchable;
use Sigmie\Mappings\NewProperties;

class Movie extends Model
{
    use Searchable;

    public function elasticsearchProperties(NewProperties $properties): void
    {
        $properties->title('title');
        $properties->name('director');
        $properties->category('genre');
        $properties->datetime('created_at');
        $properties->datetime('updated_at');
    }
}
```

For the field types you can use, see [Mappings](https://sigmie.com/docs/v2/mappings.md).

## Create the index and import rows

Create the index, then import the rows that already exist:

```bash
php artisan scout:index "App\Models\Movie"
php artisan scout:import "App\Models\Movie"
```

Pass the model class to `scout:index`. The driver creates the model to read its properties. The index name is the Scout prefix (`scout.prefix`) followed by the model's table name, for example `movies`.

## Customize every search

Define `elasticsearchSearch()` to configure the [`NewSearch`](https://sigmie.com/docs/v2/search.md) builder for every search on the model:

```php
use Sigmie\Search\NewSearch;

class Movie extends Model
{
    use Searchable;

    public function elasticsearchProperties(NewProperties $props): void
    {
        $props->title('title');
        $props->name('director');
        $props->category('genre');
    }

    public function elasticsearchSearch(NewSearch $search): void
    {
        $search->typoTolerance();
        $search->typoTolerantAttributes(['title', 'director']);
        $search->retrieve(['title', 'director']);
        $search->fields(['title', 'director']);
        $search->highlighting(
            ['title', 'director'],
            '<span class="font-bold">',
            '</span>',
        );
    }
}
```

To change one search, pass a closure to `Model::search()`. It runs after `elasticsearchSearch()`:

```php
use Sigmie\Search\NewSearch;

Movie::search('Star Wars', function (NewSearch $search) {
    $search->weight(['title' => 5]);
})->get();
```

Without `take()`, a search returns 10 results.

## Filter results

Scout's `where()`, `whereIn()`, and `whereNotIn()` become a Sigmie [filter string](https://sigmie.com/docs/v2/filter-parser.md):

```php
Movie::search('Star Wars')
    ->where('genre', 'action')
    ->whereIn('director', ['George Lucas', 'Irvin Kershner'])
    ->get();

// director:['George Lucas','Irvin Kershner'] AND genre:'action'
```

Map each field you filter on with a type that supports filters, such as `category()` or `keyword()`.

## Customize the index analysis

Define `elasticsearchIndex()` to configure the [`NewIndex`](https://sigmie.com/docs/v2/index.md) builder. Return the builder. A language builder, such as `$index->language(new English)`, is a different object, and the driver uses the one you return:

```php
use Sigmie\Index\NewIndex;

class Movie extends Model
{
    use Searchable;

    public function elasticsearchProperties(NewProperties $props): void { /* ... */ }

    public function elasticsearchIndex(NewIndex $index): NewIndex
    {
        return $index->tokenizeOnWordBoundaries()
            ->lowercase()
            ->trim()
            ->shards(3)
            ->replicas(2);
    }
}
```

Without an override, the trait tokenizes on word boundaries, lowercases, trims, and sets 1 shard and 0 replicas.

> **Note:** The driver applies the `index-settings` shards and replicas first, then calls `elasticsearchIndex()`. The default method sets 1 shard and 0 replicas, so it replaces `ELASTICSEARCH_INDEX_SHARDS` and `ELASTICSEARCH_INDEX_REPLICAS`. To use those values, override `elasticsearchIndex()` and leave out `shards()` and `replicas()`.

## Apply changed settings to an existing index

After you change `elasticsearchProperties()` or `elasticsearchIndex()`, run:

```bash
php artisan scout:sync-index-settings "App\Models\Movie"
```

The command takes the model class, not an index name. It builds a new index with the current properties and settings, copies the documents, moves the alias to it, and deletes the old index. The index must exist. Create it with `scout:index` first.

## Read the score of each result

Each model in the results carries Scout metadata from its hit:

```php
$movie = Movie::search('Star Wars')->get()->first();

$movie->scoutMetadata()['_score'];    // 32.343453
$movie->scoutMetadata()['_id'];       // "1"
$movie->scoutMetadata()['_source'];   // the indexed document
```

The models do not carry highlights. `Movie::search('Star Wars')->raw()` returns the search response as a `JsonResponse`, and each hit in it has a `highlight` key.

## Index dates

The driver formats every `DateTime` and `Carbon` value in `toSearchableArray()` as `Y-m-d\TH:i:s.u\Z`. The default `toSearchableArray()` already returns Eloquent timestamps as strings in that form, such as `2023-04-07T12:38:29.000000Z`. Map timestamp columns with `datetime()`.

If you override `toSearchableArray()`, return the `Carbon` values and let the driver format them:

```php
public function toSearchableArray(): array
{
    return [
        'title' => $this->title,
        'created_at' => $this->created_at,
        'updated_at' => $this->updated_at,
    ];
}
```

## Configuration reference

`config/elasticsearch-scout.php` has these keys:

| Key | Environment variable | Default | Description |
|-----|----------------------|---------|-------------|
| `hosts` | `ELASTICSEARCH_HOSTS` | `127.0.0.1:9200` | Hosts, separated by commas with no spaces. |
| `auth.type` | `ELASTICSEARCH_AUTH_TYPE` | `none` | `none`, `basic`, `token`, or `headers`. |
| `auth.user` | `ELASTICSEARCH_USER` | `''` | Username for `basic`. |
| `auth.password` | `ELASTICSEARCH_PASSWORD` | `''` | Password for `basic`. |
| `auth.token` | `ELASTICSEARCH_TOKEN` | `''` | Token for `token`, sent as `Authorization: Bearer <token>`. |
| `auth.headers` | none | `[]` | Headers for `headers`. |
| `guzzle_config` | none | see below | Guzzle options for the HTTP client. |
| `index-settings.shards` | `ELASTICSEARCH_INDEX_SHARDS` | `1` | Shards for new indices. |
| `index-settings.replicas` | `ELASTICSEARCH_INDEX_REPLICAS` | `2` | Replicas for new indices. |

The default `guzzle_config` is:

```php
'guzzle_config' => [
    'allow_redirects' => false,
    'http_errors' => false,
    'connect_timeout' => 15,
],
```

Add any [Guzzle option](https://sigmie.com/docs/v2/connection.md#configuration-options) to it, such as `'timeout' => 30`.

## Configure authentication

For basic auth:

```ini
ELASTICSEARCH_AUTH_TYPE=basic
ELASTICSEARCH_USER=elastic
ELASTICSEARCH_PASSWORD=your-password
```

For a bearer token:

```ini
ELASTICSEARCH_AUTH_TYPE=token
ELASTICSEARCH_TOKEN=your-token-here
```

For an API key or other custom headers, set the type to `headers`:

```ini
ELASTICSEARCH_AUTH_TYPE=headers
```

Then fill `auth.headers` in `config/elasticsearch-scout.php`:

```php
'headers' => [
    'Authorization' => 'ApiKey ' . env('ELASTICSEARCH_API_KEY'),
],
```

The driver sends `auth.headers` only when the type is `headers`.

## See also

- [Search](https://sigmie.com/docs/v2/search.md) for every option in `elasticsearchSearch()`.
- [Mappings](https://sigmie.com/docs/v2/mappings.md) for the field types in `elasticsearchProperties()`.
- [Indices](https://sigmie.com/docs/v2/index.md) for the analysis options in `elasticsearchIndex()`.
- [Laravel AI SDK](https://sigmie.com/docs/v2/laravel-ai.md) to expose Scout models as AI agent tools.


---

<!-- source: https://sigmie.com/docs/v2/laravel-ai -->

# Laravel AI SDK

Sigmie exposes an index as a set of [Laravel AI SDK](https://laravel.com/docs/ai-sdk) tools. An agent with these tools searches, filters, sorts, facets, and reads documents in the index. The tools build their descriptions from your property definitions. They require the `laravel/ai` package.

## Register the tools in an agent

Add the `AsTool` trait to a `SigmieIndex` subclass:

```php
use Sigmie\AI\AsTool;
use Sigmie\Mappings\NewProperties;
use Sigmie\SigmieIndex;

class ProductIndex extends SigmieIndex
{
    use AsTool;

    public function name(): string
    {
        return 'products';
    }

    public function properties(): NewProperties
    {
        $props = new NewProperties;
        $props->name('name');
        $props->category('brand');
        $props->number('price');
        $props->bool('in_stock');
        return $props;
    }
}
```

Return the index's `tools()` from the agent:

```php
use Laravel\Ai\Contracts\Agent;
use Laravel\Ai\Contracts\HasTools;
use Laravel\Ai\Promptable;

class ShoppingAssistant implements Agent, HasTools
{
    use Promptable;

    public function instructions(): string
    {
        return 'You help users find products in our catalog.';
    }

    public function tools(): array
    {
        return [
            ...app(ProductIndex::class)->tools(),
        ];
    }
}
```

To give the agent the search tool alone, construct it directly. This does not need the trait:

```php
use Sigmie\AI\SigmieIndexTool;

new SigmieIndexTool(app(ProductIndex::class));
```

## The tool suite

`tools(string $baseFilter = '')` returns six tools, in this order:

| # | Class | `name()` | Purpose |
|---|-------|----------|---------|
| 0 | `SigmieIndexTool` | not defined | Query, filter, sort, facet, and paginate. |
| 1 | `SigmieFilterValuesTool` | `discover_filter_values` | List the values of one facetable field. |
| 2 | `SigmieSampleDocumentsTool` | `sample_documents` | Return random documents, so the agent sees the data shape. |
| 3 | `SigmieGetDocumentsTool` | `get_documents` | Return documents by `_id`. |
| 4 | `SigmieIndexSchemaTool` | `describe_index` | Return the field list, types, and filter syntax as structured data. |
| 5 | `SigmieAnalyticsTool` | `analytics` | Compute one dashboard widget per call. See [Analytics](https://sigmie.com/docs/v2/analytics.md). |

The tool descriptions refer to the search tool as `search_index`.

## Base filters

A base filter limits every query the agent runs. Use it for multi-tenancy or per-user access. The agent does not choose the base filter and does not see it in any tool description.

```php
app(OrderIndex::class)->tools("user_id:{$user->id}");

new SigmieIndexTool(app(OrderIndex::class), baseFilters: "user_id:{$user->id}");
```

Sigmie parses the base filter on its own and adds it as a separate filter clause. It parses the agent's `filters` string on its own too, so the two strings never join as text:

```
base filter:  user_id:3                              -> hard filter clause
agent filter: status:'shipped' OR status:'delivered' -> second clause
match:        documents that pass both clauses
```

An agent filter with unbalanced parentheses, such as `status:'x') OR (user_id:4`, cannot form an `OR` that escapes the scope. The tool rejects it. `handle()` returns an `{"error": ...}` the agent can correct from, and `result()` throws a `ParseException`.

Every tool that reads documents applies the base filter, including `sample_documents` and `get_documents`. `describe_index` reads only the mapping, so it takes no base filter. Sampling draws only from in-scope documents. An out-of-scope id gives the same result as a missing id:

```php
use Laravel\Ai\Tools\Request;

[, , $sample, $get] = app(OrderIndex::class)->tools('user_id:3');

$get->handle(new Request(['ids' => ['order-of-user-3', 'order-of-user-4']]));
// [{"_id": "order-of-user-3", ...}]  (order-of-user-4 is omitted, like a missing id)
```

> **Warning:** A malformed base filter throws a `ParseException` on every call. It never falls back to matching all documents.

## Output fields

`SigmieIndexTool`, `sample_documents`, and `get_documents` return every source field by default. For an index with large fields, such as a full article body, ten hits can fill the agent's context.

The agent passes `fields`, a comma-separated list of source fields. Dotted paths reach nested fields:

```json
{"query": "solar", "fields": "title,author,sections.text"}
```

Override `toolFields()` to set the fields the tools return when the agent passes `fields: null`. An empty list, the default, returns every field:

```php
class ArticleIndex extends SigmieIndex
{
    use AsTool;

    public function properties(): NewProperties
    {
        $props = new NewProperties;
        $props->title('title');
        $props->category('author');
        $props->date('published_at');
        $props->longText('body');                // full article body
        $props->nested('sections', function (NewProperties $props) {
            $props->keyword('type');
            $props->text('text');                // one section
        });
        $props->nested('comments', function (NewProperties $props) {
            $props->text('text');
        });

        return $props;
    }

    public function toolFields(): array
    {
        return ['title', 'author', 'published_at'];
    }
}
```

The agent can still request a field outside the default, such as `fields: "body"`. The `fields` parameter description names the default, so the agent knows what it gets. [Private fields](#private-fields) always win. A field from `exceptFromTools()` never returns, even when the agent requests it.

## Matching nested items

A search for "solar" matches the article, but the hit alone does not say which of its `sections` matched. The `matches` argument of `SigmieIndexTool` returns those sections. It lists the nested paths whose matching items the agent wants back. Sigmie runs these as Elasticsearch inner hits, described in [Search](https://sigmie.com/docs/v2/search.md#matching-nested-items-inner-hits).

With `fields: null`, the index default above keeps the body and the full section array out:

```json
{"query": "solar", "fields": null, "matches": "sections.text"}
```

Each hit gets `_matches` next to its source fields:

```json
{
    "total": 1,
    "hits": [
        {
            "_id": "article-1",
            "title": "Energy at home",
            "author": "Jane Doe",
            "published_at": "2024-05-02",
            "_matches": {
                "sections": {
                    "total": 4,
                    "items": [
                        {"_offset": 1, "text": "The solar panels were installed."},
                        {"_offset": 2, "text": "The solar inverter was replaced."},
                        {"_offset": 4, "text": "A solar battery stores the surplus."},
                        {"_offset": 3, "text": "Solar power pays off in six years."}
                    ]
                }
            }
        }
    ]
}
```

| Key | Holds |
|-----|-------|
| `_matches.sections.total` | How many sections of this article matched the query or a nested filter. |
| `_matches.sections.items` | The matching sections, best match first, up to 100. |
| `_offset` | The section's position in the `sections` array. |

`matches: "sections"` returns whole sections. `matches: "sections.text sections.type"` returns those fields of each section. Spaces or commas separate entries.

Add `:size` to an entry to cap that path's items, the same way `facets` takes `field:size`. Each path has its own size, from 1 to 100. The default is 100:

```json
{"query": "solar", "fields": null, "matches": "sections.text:2 comments"}
```

Here `_matches.sections` holds 2 of the 4 matching sections with `total: 4`, and `_matches.comments` holds every matching comment. An invalid size, such as `sections.text:0` or `sections.text:500`, returns an error the agent can correct from:

```json
{"error": "Invalid matches entry 'sections.text:0'. Use a nested path, optionally with ':size' from 1 to 100, e.g. 'sections.text:5'. Check the field names and the filter/sort syntax in this tool's description, then try again."}
```

A nested filter such as `sections:{type:'conclusion'}` also produces matches. A section that matches both the query and the filter appears once. With `matches: null`, the default, hits have no `_matches` key. [Private fields](#private-fields) never appear in matches.

## Private fields

Override `exceptFromTools()` to keep fields out of every document the tools return. Dotted paths reach nested fields:

```php
class CustomerIndex extends SigmieIndex
{
    use AsTool;

    public function properties(): NewProperties
    {
        $props = new NewProperties;
        $props->name('name');
        $props->nested('contacts', function (NewProperties $props) {
            $props->keyword('name');
            $props->keyword('email');
        });

        return $props;
    }

    public function exceptFromTools(): array
    {
        return ['contacts.email'];
    }
}
```

`SigmieIndexTool`, `sample_documents`, `get_documents`, and the `analytics` tool's `table` widget and `include_hits` rows omit these fields. Elasticsearch drops them from `_source` before the response leaves the cluster. The exclusion wins over the agent's own `fields` and `hit_fields`, so `hit_fields: "contacts"` returns `contacts.name` only.

The agent can still filter on a private field:

```
contacts:{email:'jane@example.com'}
```

Every tool refuses to list, facet, group, measure, or sort by a private field or any field below it. This covers `discover_filter_values`, the search `facets` and `sort`, and every `analytics` argument that names a field, such as `group_by`, `group_by_fields`, `row_field`, `field`, `sort`, and `hit_sort`. `handle()` returns an error the agent can correct from, and `result()` throws an `InvalidArgumentException`:

```json
{"error": "Field contacts.email is private and cannot be listed, grouped, faceted or sorted; you can still filter on it. ..."}
```

The tool descriptions and `describe_index` mark these fields as filter only, so the agent does not try. Your own code keeps full access. `facets()` on a search and `analytics()` on the index ignore `exceptFromTools()`.

## Search tool description

`SigmieIndexTool` builds its description from the index properties. For these properties:

```php
$props->keyword('brand');
$props->number('price');
$props->bool('in_stock');
$props->date('created_at');
```

The agent sees this description. The last lines are shortened here:

```
Search the 'products' index.

Available fields:
- brand [keyword] (sortable, facetable): brand:'value' brand:['a','b'] brand:val*
- price [number] (sortable, facetable): price>n price<=n price:min..max
- in_stock [boolean] (sortable): in_stock:true in_stock:false
- created_at [date] (sortable): created_at>'2024-01-01' created_at<'2024-12-31'

Filter operators: AND, OR, AND NOT
Negation: NOT field:'value'
Grouping: (field:'a' OR field:'b') AND other>10
Exists check: field:*
Sort: space-separated list of 'field:asc' or 'field:desc' ...
Geo sort: field[lat,lon]:km:asc
Facets: field1 field2:20 (space-separated, optional :size for keywords or :interval for numbers)
Matching: equality filters on text/keyword fields are exact and CASE-SENSITIVE ...
Output: each hit holds `_id` plus the source fields in `fields` (null uses the index default).
Matches: to see WHICH items of a nested field matched the query or a nested filter, ...
Discovering valid values: if you do not know a field's valid values, call discover_filter_values ...
Schema: call describe_index for the full structured field list, types and filter syntax.
```

A field's `description` from the mapping follows its line. A private field shows `(filter only)` instead of its capabilities. The full syntax lives in [Filter parser](https://sigmie.com/docs/v2/filter-parser.md), [Sort parser](https://sigmie.com/docs/v2/sort-parser.md), and [Facets](https://sigmie.com/docs/v2/facets.md).

## Search tool parameters

Every parameter except `query` is nullable and listed as required, so the schema is valid under OpenAI strict function calling. The agent passes `null` for a parameter it does not use.

| Parameter | Type | Description |
|-----------|------|-------------|
| `query` | string | Search query text. |
| `filters` | string | Filter expression. See [Filter parser](https://sigmie.com/docs/v2/filter-parser.md). |
| `sort` | string | Sort expression. See [Sort parser](https://sigmie.com/docs/v2/sort-parser.md). |
| `facets` | string | Space-separated facet fields. See [Facets](https://sigmie.com/docs/v2/facets.md). |
| `facet_filters` | string | Active facet filter values. |
| `per_page` | int | Results per page. Default 10. |
| `page` | int | Page number. Default 1. |
| `fields` | string | Comma-separated source fields to return. `null` uses `toolFields()`. |
| `matches` | string | Nested paths whose matching items return in `_matches`. Spaces or commas separate entries, each optionally `path:size` (1 to 100, default 100). `null` returns none. |

The result holds `total` and `hits`. Each hit holds `_id`, the selected source fields, and `_matches` when requested. When the agent requests facets, the result also holds a `facets` key with the facet data.

## Companion tool parameters

| Tool | Parameter | Description |
|------|-----------|-------------|
| `discover_filter_values` | `field` | Facetable field to list values for. Required. |
| | `filters` | Filter expression that narrows the values. |
| | `limit` | Maximum values to return. Default 20. |
| `sample_documents` | `limit` | Number of documents, 1 to 20. Default 5. |
| | `fields` | Same as the search tool's `fields`. |
| `get_documents` | `ids` | Document ids, up to 100. Required. Results keep the order of `ids`. |
| | `fields` | Same as the search tool's `fields`. |
| `describe_index` | none | Returns `index`, `fields`, `filter_syntax`, and `notes`. |

For the `analytics` widgets and their arguments, see [Analytics](https://sigmie.com/docs/v2/analytics.md).

## Discovered values

`discover_filter_values` returns at most `limit` values. When more values exist, `truncated` is `true` and `other_documents` counts the documents in the values left out:

```json
{
    "field": "color",
    "values": { "red": 12, "blue": 7 },
    "truncated": true,
    "other_documents": 5
}
```

Here `limit` was 2, and 5 more documents have other colors. The tool description tells the agent to raise `limit` or narrow the list with `filters`, such as `"color:b*"`, before it treats the list as complete.

Numeric and date fields return min and max, and always report `truncated: false`. A date field returns the period it covers as date strings, and `filters` narrow that period:

```json
{
    "field": "published_at",
    "values": { "count": 3, "min": "2023-01-04T00:00:00.000Z", "max": "2025-11-30T00:00:00.000Z" },
    "truncated": false,
    "other_documents": 0
}
```

## Errors

The tools parse the agent's `filters`, `facets`, and `facet_filters` strictly. An unknown field or an expression that does not parse returns an `error` result the agent can read and correct. It does not return an empty result:

```
filters: color:'red'    (no "color" field)

result: {"error": "Field color does not exist. Check the field names and the filter/sort syntax in this tool's description, then try again."}
```

`discover_filter_values` applies the same rule to its `field` and `filters`.

Only the agent-facing `handle()` returns errors as JSON. The `result()` methods return arrays and throw, so your own code keeps normal exception handling:

```php
use Laravel\Ai\Tools\Request;
use Sigmie\AI\SigmieFilterValuesTool;
use Sigmie\Parse\ParseException;

try {
    (new SigmieFilterValuesTool($index))->result(new Request(['field' => 'nope']));
} catch (ParseException $e) {
    // "Field nope does not exist."
}
```

## See also

- [Filter parser](https://sigmie.com/docs/v2/filter-parser.md) for every filter operator.
- [Sort parser](https://sigmie.com/docs/v2/sort-parser.md) for sort expression syntax.
- [Facets](https://sigmie.com/docs/v2/facets.md) for facet behavior and structure.
- [Analytics](https://sigmie.com/docs/v2/analytics.md) for the widgets behind the `analytics` tool.
- [Search](https://sigmie.com/docs/v2/search.md) for the search builder the tools use.
- [MCP server](https://sigmie.com/docs/v2/mcp.md) to connect a coding assistant to the Sigmie documentation.


---

<!-- source: https://sigmie.com/docs/v2/mcp -->

# MCP Server

Sigmie runs a remote [Model Context Protocol (MCP)](https://modelcontextprotocol.io) server at `https://sigmie.com/mcp`. Connect your coding assistant to it, and the assistant searches and reads the Sigmie documentation while it writes code. The server uses the Streamable HTTP transport.

To expose your own indices to an agent, use the [Laravel AI SDK](https://sigmie.com/docs/v2/laravel-ai.md) tools instead.

## Add the server to Claude Code

To add the server to one project, create `.mcp.json` in the project root:

```json
{
  "mcpServers": {
    "sigmie-docs": {
      "type": "http",
      "url": "https://sigmie.com/mcp"
    }
  }
}
```

To add the server to every project, run:

```bash
claude mcp add --transport http --scope user sigmie-docs https://sigmie.com/mcp
```

The command writes the same entry to `~/.claude.json`.

## Add the server to Cursor

Add this entry to Cursor's MCP settings:

```json
{
  "mcpServers": {
    "sigmie-docs": {
      "url": "https://sigmie.com/mcp"
    }
  }
}
```

For another MCP client, point an HTTP server entry at `https://sigmie.com/mcp`.

## Check the connection

Restart the assistant. It now lists three tools from `sigmie-docs`:

| Tool | Call | Returns |
|------|------|---------|
| `search_docs` | `search_docs({ query: "how to configure semantic search" })` | The 10 best matching sections, each with title, page slug, version, URL, and content. |
| `read_doc` | `read_doc({ page: "search", version: "v2" })` | The full Markdown of one page, including its code examples. |
| `list_docs` | `list_docs({ version: "v2" })` | Every page slug for a version. |

`search_docs` runs a hybrid query, keyword and vector, against an Elasticsearch index of the documentation sections. The assistant usually calls `search_docs` first, then `read_doc` on the page it needs.

## See also

- [Laravel AI SDK](https://sigmie.com/docs/v2/laravel-ai.md) to expose your own Sigmie indices as agent tools.
- [Semantic search](https://sigmie.com/docs/v2/semantic-search.md) for the vector search the docs server uses.
- [Retrieval and Agents](https://sigmie.com/docs/v2/rag.md) to build retrieval-augmented generation with Sigmie.


---

<!-- source: https://sigmie.com/docs/v2/packages -->

# Packages

`sigmie/sigmie` is one package that contains the whole library. Install it with Composer:

```bash
composer require sigmie/sigmie
```

## Contents of `sigmie/sigmie`

The main namespaces in the package:

| Namespace | Contents |
|-----------|----------|
| `Sigmie\Base` | Connection, HTTP requests, and the Elasticsearch and OpenSearch drivers. |
| `Sigmie\Index` | `NewIndex`, analysis, aliases, updates, and rebuilds. |
| `Sigmie\Mappings` | `NewProperties` and the field types. |
| `Sigmie\Document` | `Document`, `AliveCollection`, and collection hooks. |
| `Sigmie\Query` | The low-level query builder, `NewQuery`, and aggregations. |
| `Sigmie\Search` | The search builder, `NewSearch`, and multi-search, templates, and recommendations. |
| `Sigmie\Parse` | The filter, sort, and facet string parsers. |
| `Sigmie\Languages` | English, German, and Greek analysis. |
| `Sigmie\AI` | Embedding, reranking, and CLIP API clients, and agent tools. |
| `Sigmie\Analytics` | Analytics queries. |
| `Sigmie\Contracts` | The `Package` interface for [extensions](https://sigmie.com/docs/v2/extending.md). |

## Related packages

| Package | Namespace | Purpose |
|---------|-----------|---------|
| `sigmie/http` | `Sigmie\Http` | `JSONClient`, the Guzzle-based HTTP client. `sigmie/sigmie` requires it, so Composer installs it for you. |
| `sigmie/testing` | `Sigmie\Testing` | PHPUnit `TestCase`, `Assert`, and index assertions. The `sigmie/sigmie` archive leaves out its test helpers, so install this package to use them. |
| `sigmie/elasticsearch-scout` | `Sigmie\ElasticsearchScout` | Laravel Scout driver. See [Laravel Scout](https://sigmie.com/docs/v2/laravel-scout.md). |

```bash
composer require --dev sigmie/testing
composer require sigmie/elasticsearch-scout
```

## Legacy packages

These packages are older splits of the library. `sigmie/sigmie` contains their code, so do not install them next to it.

| Package | Last version | Status |
|---------|--------------|--------|
| `sigmie/base` | 0.38.0 | Abandoned on Packagist. |
| `sigmie/shared` | 0.38.0 | Abandoned on Packagist. |
| `sigmie/mappings` | 0.38.0 | Abandoned on Packagist. |
| `sigmie/query` | 0.38.0 | Abandoned on Packagist. |
| `sigmie/search` | 0.38.0 | Abandoned on Packagist. |
| `sigmie/parse` | 0.38.0 | Abandoned on Packagist. |
| `sigmie/english` | 1.0.0 | Uses the old `Sigmie\English` namespace. Use `Sigmie\Languages\English` in `sigmie/sigmie`. |
| `sigmie/german` | 1.0.0 | Uses the old `Sigmie\German` namespace. Use `Sigmie\Languages\German` in `sigmie/sigmie`. |
| `sigmie/greek` | 1.0.0 | Uses the old `Sigmie\Greek` namespace. Use `Sigmie\Languages\Greek` in `sigmie/sigmie`. |


---

<!-- source: https://sigmie.com/docs/v2/rag -->

# Retrieval and Agents

This guide builds a retrieval-augmented generation (RAG) flow: retrieve documents with Sigmie, rerank them, and pass the best ones to a language model. Sigmie handles retrieval and reranking. It ships no LLM client, prompt builder, or single "question to answer" API, so the generation step is your own code.

| Step | Owner | API |
|------|-------|-----|
| Retrieve | Sigmie | `newSearch()` with `->semantic()` |
| Rerank | Sigmie | `$response->rerank(...)` with a registered `RerankApi` |
| Generate | Your code | Any HTTP client, vendor SDK, or the [Laravel AI SDK](https://sigmie.com/docs/v2/laravel-ai.md) |

## Before you start

You need an index with a semantic field. [Semantic search](https://sigmie.com/docs/v2/semantic-search.md) shows how to define one and index documents. The examples below search a `docs` index whose `content` field holds the text.

## Register the APIs

Register an embeddings API for semantic retrieval and a rerank API for reranking:

```php
use Sigmie\AI\APIs\CohereRerankApi;
use Sigmie\AI\APIs\OpenAIEmbeddingsApi;

$sigmie->registerApi('embeddings', new OpenAIEmbeddingsApi('sk-...'));
$sigmie->registerApi('reranker', new CohereRerankApi('co-...'));
```

## Retrieve, rerank, and generate

```php
// 1. Retrieve 20 candidates.
$response = $sigmie->newSearch('docs')
    ->properties($props)
    ->semantic()
    ->queryString('What is your return policy?')
    ->size(20)
    ->get();

// 2. Keep the 5 best by reranker score.
$top5 = $response->rerank('reranker', ['content'], topK: 5);

// 3. Generate with your own client. $yourOpenAiClient is a placeholder.
$context = collect($top5)->pluck('_source.content')->implode("\n\n");

$answer = $yourOpenAiClient->chat([
    ['role' => 'system', 'content' => 'Answer using only the provided context.'],
    ['role' => 'user', 'content' => "Context:\n{$context}\n\nQuestion: What is your return policy?"],
]);
```

`rerank()` returns an array of `RerankedHit` objects, best first. Each holds the original `_id` and `_source`, plus `_rerank_score`.

## Adjust the rerank step

`rerank()` takes a registered API name or a `RerankApi` instance:

```php
rerank(
    RerankApi|string $reranker,
    array $fields,
    ?string $query = null,
    ?int $topK = null,
): array;
```

- To send more than one field to the reranker, list them: `rerank('reranker', ['title', 'content'])`. Sigmie sends each hit's fields as a YAML snippet.
- To rerank against a different text than the search query, pass `$query`: `rerank('reranker', ['content'], 'return policy')`. By default it uses the search's query string.
- To change how many hits come back, pass `topK`. By default it uses the search's `size`, or 10 when the size is 0.

With no hits or an empty query, `rerank()` calls no API and returns the hits unchanged. A name that is not a registered `RerankApi` throws an `InvalidArgumentException`. To rerank hits outside a search response, use `Sigmie\Search\NewRerank` directly.

## See also

- [Semantic search](https://sigmie.com/docs/v2/semantic-search.md) for embeddings, providers, and similarity.
- [Recommendations](https://sigmie.com/docs/v2/recommendations.md) for RRF and MMR in similar-item retrieval.
- [Magic Tags](https://sigmie.com/docs/v2/magic-tags.md) for taxonomy tags backed by embeddings.
- [Laravel AI SDK](https://sigmie.com/docs/v2/laravel-ai.md) to let an agent search an index through tools.
- [MCP server](https://sigmie.com/docs/v2/mcp.md) to connect a coding assistant to the Sigmie documentation.


---

<!-- source: https://sigmie.com/docs/v2/extending -->

# Extending Sigmie

This page shows how to write a package that extends Sigmie. A package can add two things:

- Field-type methods on `NewProperties`, through macros.
- Collection hooks, which run code around `merge()` and `add()` when documents are indexed.

[Magic Tags](https://sigmie.com/docs/v2/magic-tags.md) is a package built this way. It adds a `magicTags()` field type and a hook that calls an LLM and writes tags to a sidecar index. The examples on this page use it.

## Register a package

Call `extend()` on a `Sigmie` instance:

```php
use Sigmie\Sigmie;
use Vendor\MagicTags\MagicTagsPackage;

$sigmie = new Sigmie($connection);
$sigmie->extend(new MagicTagsPackage());
```

`extend()` calls the package's `register()` method at once. Each call to `extend()` calls `register()` again.

Hooks belong to the instance you call `extend()` on. Two `Sigmie` instances in one PHP process can have different hooks.

> **Note:** Macros are not per instance. `NewProperties::macro()` stores them in a static array for the whole process. In tests that need isolation, call `NewProperties::flushMacros()` between cases.

## Implement the `Package` interface

A package implements `Sigmie\Contracts\Package`. Register macros and hooks in `register()`:

```php
namespace Vendor\MagicTags;

use Sigmie\Contracts\Package;
use Sigmie\Mappings\NewProperties;
use Sigmie\Sigmie;
use Vendor\MagicTags\Types\MagicTags;

class MagicTagsPackage implements Package
{
    public function register(Sigmie $sigmie): void
    {
        NewProperties::macro('magicTags', function (string $name, string $fromField): MagicTags {
            $field = new MagicTags($name, $fromField);
            $this->add($field);

            return $field;
        });

        $sigmie->addCollectionHook(new MagicTagsCollectionHook());
    }
}
```

## Add a field type with a macro

`NewProperties::macro()` takes a name and a closure. Sigmie binds the closure to the `NewProperties` instance, so `$this` is the builder. Add the field with `$this->add()` and return it, so callers can chain methods on it:

```php
use Sigmie\Mappings\NewProperties;
use Vendor\MagicTags\Types\MagicTags;

NewProperties::macro('magicTags', function (string $name, string $fromField): MagicTags {
    $field = new MagicTags($name, $fromField);
    $this->add($field);

    return $field;
});
```

After registration, callers use the macro like a built-in type:

```php
$props = new NewProperties;
$props->text('content')->semantic(api: 'embeddings', accuracy: 1, dimensions: 1024);
$props->magicTags('topic', fromField: 'content')
    ->api('llm')
    ->embeddingsApi('embeddings');
```

A call to a method with no macro throws `BadMethodCallException`.

## Add a collection hook

A hook implements `Sigmie\Document\Contracts\CollectionHook`. Register it on the same `Sigmie` instance:

```php
$sigmie->addCollectionHook(new MagicTagsCollectionHook());
```

The interface has four methods:

```php
namespace Vendor\MagicTags;

use Sigmie\Document\Contracts\CollectionHook;
use Sigmie\Mappings\Properties;
use Sigmie\Sigmie;
use Vendor\MagicTags\Types\MagicTags;

class MagicTagsCollectionHook implements CollectionHook
{
    public function shouldRun(Properties $properties): bool
    {
        return $properties->fieldsOfType(MagicTags::class)->isNotEmpty();
    }

    public function beforeBatch(
        string $indexName,
        Sigmie $sigmie,
        Properties $properties,
        array $apis
    ): void {
        // Create the sidecar index if it does not exist.
    }

    public function processBatch(
        array $documents,
        Properties $properties,
        array $apis
    ): array {
        // Call the LLM and add tags. Return every document.
        return $documents;
    }

    public function afterBatch(
        array $documents,
        string $indexName,
        Sigmie $sigmie,
        Properties $properties,
        array $apis
    ): void {
        // Write (magic_field_path, tag) rows to the sidecar index.
    }
}
```

### When each method runs

For `merge($documents)`, the collection calls the hooks in this order:

1. `shouldRun()` on each hook. Hooks that return `false` do not run for this batch.
2. `beforeBatch()` on each hook, once.
3. Sigmie formats dates, validates fields, and adds embeddings.
4. `processBatch()` on each hook, in the order you registered them. Each hook gets the documents the previous hook returned.
5. Sigmie writes the documents to the index.
6. `afterBatch()` on each hook, with the documents that were written.

`add($document)` runs the same steps for one document. `processBatch()` and `afterBatch()` get an array that holds only that document.

Hooks run on collections from `$sigmie->collect()`, and on the collections that `rebuild()` and `rebuilding()` give you. See [Indices](https://sigmie.com/docs/v2/index.md) for rebuilds.

### Run only on matching indices

`shouldRun()` gets the collection's properties. Return `true` only when they contain your field type. Without this check, the hook runs for every collection on the instance, including indices your package does not own.

### Use the `$sigmie` argument

The `$sigmie` that `beforeBatch()` and `afterBatch()` get is a new instance on the same connection. It has no registered APIs and no hooks, so writes to a sidecar index through it do not run hooks.

### Read the `$apis` map

`processBatch()` and `afterBatch()` get the collection's APIs as a map of name to instance. `collect()` starts the map with the APIs from `Sigmie::registerApi()`. A call to `apis()` on the collection replaces the whole map.

`registerApi()` accepts only `EmbeddingsApi` and `RerankApi` instances. To give a hook an LLM client, inject it in the hook's constructor, or pass it to the collection's `apis()`:

```php
$embeddings = $apis['embeds'] ?? null;    // EmbeddingsApi
$rerank = $apis['my-rerank'] ?? null;     // RerankApi
$llm = $this->llmClient;                  // injected in the constructor
```

## Index without hooks

When the documents already hold the values a hook would add, call `withoutHooks()`. The collection then skips every hook:

```php
$sigmie->collect('kb')->withoutHooks()->merge($documents);
```

## Use the package in an application

```php
use Sigmie\Mappings\NewProperties;
use Sigmie\Sigmie;
use Vendor\MagicTags\MagicTagsPackage;

$sigmie = new Sigmie($connection);
$sigmie->extend(new MagicTagsPackage());

$props = new NewProperties;
$props->text('content')->semantic(api: 'embeds', accuracy: 1, dimensions: 1024);
$props->magicTags('topic', fromField: 'content')->api('llm')->embeddingsApi('embeds');

$sigmie->collect('kb', refresh: true)
    ->properties($props)
    ->apis([
        'llm' => $llmApi,
        'embeds' => $embeddingsApi,
    ])
    ->merge([/* documents */]);
```

Each `merge()` or `add()` on a collection with a `MagicTags` field now runs the hook. This applies only to collections from the instance you called `extend()` on.

## See also

- [Magic Tags](https://sigmie.com/docs/v2/magic-tags.md), a package built on this API.
- [Mappings](https://sigmie.com/docs/v2/mappings.md) for field types and properties.
- [Documents](https://sigmie.com/docs/v2/document.md) for the collection lifecycle.
