Semantic Search

Build semantic vector search with Sigmie — semantic fields, accuracy levels, similarity metrics, and embeddings from OpenAI, Cohere, or Infinity.

On this page

Semantic search matches documents by meaning, not just keywords. “Portable computer for work” can match documents containing “laptop”, “notebook”, or “MacBook” — none of which share a word with the query.

Sigmie does this by:

  1. Generating vector embeddings from your text at index time.
  2. Generating an embedding for the query at search time.
  3. Returning the documents whose vectors are most similar.

You bring an embeddings API (OpenAI, Cohere, Voyage, Infinity, or anything implementing EmbeddingsApi) and register it with the client:

use Sigmie\AI\APIs\OpenAIEmbeddingsApi;
 
$sigmie->registerApi('embeddings', new OpenAIEmbeddingsApi('sk-...'));

The name 'embeddings' is yours to choose — refer to it from your field definitions.

Define a semantic field

use Sigmie\Mappings\NewProperties;
 
$props = new NewProperties;
$props->title('title')->semantic(api: 'embeddings', dimensions: 1536);
$props->text('description')->semantic(api: 'embeddings', dimensions: 1536);
 
$sigmie->newIndex('articles')->properties($props)->create();

Match the dimensions to the model you registered. OpenAI’s text-embedding-3-small outputs 1536-dim vectors; Infinity’s bge-small-en-v1.5 outputs 384-dim.

Index documents

When a property has ->semantic(), Sigmie generates embeddings automatically as documents flow through merge() and add():

use Sigmie\Document\Document;
 
$sigmie->collect('articles', refresh: true)
    ->properties($props)
    ->merge([
        new Document([
            'title' => 'Introduction to Machine Learning',
            'description' => 'A primer on supervised and unsupervised learning.',
        ]),
        new Document([
            'title' => 'Deep Learning Fundamentals',
            'description' => 'Neural networks form the basis of deep learning.',
        ]),
    ]);

Search

Enable semantic matching with ->semantic():

$response = $sigmie->newSearch('articles')
    ->properties($props)
    ->semantic()
    ->queryString('artificial intelligence basics')
    ->get();

By default this combines semantic and keyword matching. Documents matched by both rank higher than documents matched by only one.

Pure semantic search

Drop keyword matching entirely:

$sigmie->newSearch('articles')
    ->properties($props)
    ->semantic()
    ->disableKeywordSearch()
    ->queryString('machine learning algorithms')
    ->get();

Score multipliers

Bias the blend between keyword and semantic scores:

$sigmie->newSearch('articles')
    ->properties($props)
    ->semantic()
    ->textScoreMultiplier(1.0)
    ->semanticScoreMultiplier(2.0)        // emphasize semantic matches
    ->queryString('quantum computing')
    ->get();

Embedding providers

Sigmie ships clients for several providers — all implement Sigmie\AI\Contracts\EmbeddingsApi:

use Sigmie\AI\APIs\OpenAIEmbeddingsApi;
use Sigmie\AI\APIs\CohereEmbeddingsApi;
use Sigmie\AI\APIs\VoyageEmbeddingsApi;
use Sigmie\AI\APIs\InfinityEmbeddingsApi;
 
$sigmie->registerApi('embeddings', new OpenAIEmbeddingsApi('sk-...'));
$sigmie->registerApi('embeddings', new CohereEmbeddingsApi('co-...'));
$sigmie->registerApi('embeddings', new VoyageEmbeddingsApi('pa-...'));
 
// Local Infinity service (see Docker docs)
$sigmie->registerApi('embeddings', new InfinityEmbeddingsApi(
    baseUrl: 'http://localhost:7997',
    model: 'BAAI/bge-small-en-v1.5',
));

Custom provider

Implement EmbeddingsApi:

use Sigmie\AI\Contracts\EmbeddingsApi;
use GuzzleHttp\Promise\Promise;
 
class MyEmbeddings implements EmbeddingsApi
{
    public function embed(string $text, int $dimensions): array { /* ... */ }
    public function batchEmbed(array $payload): array { /* ... */ }
    public function promiseEmbed(string $text, int $dimensions): Promise { /* ... */ }
    public function model(): string { /* ... */ }
}
 
$sigmie->registerApi('embeddings', new MyEmbeddings());

Accuracy

The accuracy parameter controls the HNSW index parameters under the hood. Higher accuracy means better recall at the cost of more memory and slower indexing:

$props->text('content')->semantic(api: 'embeddings', dimensions: 512, accuracy: 1);
// Fast: m=16, ef_construction=80
 
$props->text('content')->semantic(api: 'embeddings', dimensions: 512, accuracy: 3);
// Balanced (default): m=64, ef_construction=300
 
$props->text('content')->semantic(api: 'embeddings', dimensions: 512, accuracy: 5);
// High quality: m=128, ef_construction=800
 
$props->text('content')->semantic(api: 'embeddings', dimensions: 512, accuracy: 7);
// Script-score: exact vectors, slowest, highest quality

Reach for higher accuracy on important fields (titles, primary content). Use lower accuracy on long, less-critical fields (tags, supporting text).

Similarity functions

use Sigmie\Enums\VectorSimilarity;
 
$props->text('content')->semantic(
    api: 'embeddings',
    dimensions: 512,
    similarity: VectorSimilarity::Cosine,            // default
);
 
// Other options:
VectorSimilarity::DotProduct;
VectorSimilarity::Euclidean;
VectorSimilarity::MaxInnerProduct;
  • Cosine — standard for text similarity, handles different lengths.
  • Dot product — efficient when your vectors are pre-normalized.
  • Euclidean — distance-based, sensitive to magnitude.
  • Max inner product — optimized for IP-similarity workloads.

Multiple vectors per field

Index the same text with different similarity functions or accuracies:

$props->text('job_description')
    ->semantic(
        api: 'embeddings',
        accuracy: 3,
        dimensions: 512,
        similarity: VectorSimilarity::Cosine,
    )
    ->semantic(
        api: 'embeddings',
        accuracy: 5,
        dimensions: 512,
        similarity: VectorSimilarity::Euclidean,
    );

Field-specific semantic search

Restrict semantic matching to specific fields:

$sigmie->newSearch('articles')
    ->properties($props)
    ->semantic()
    ->fields(['title'])
    ->queryString('deep learning neural networks')
    ->get();

Working with arrays

Semantic fields work with array values — each entry is embedded independently:

new Document([
    'experience' => [
        'Artist',
        'Graphic Design',
        'Creative Director',
    ],
]);
 
// "drawing illustration" matches "Artist" semantically
$sigmie->newSearch('professionals')
    ->semantic()
    ->properties($props)
    ->queryString('drawing illustration')
    ->get();

Pre-computed embeddings

To skip Sigmie’s embedding pipeline (for backfills or batched offline embedding), include vectors directly in the document:

new Document([
    'title' => 'AI Research Paper',
    'content' => 'Artificial intelligence has evolved significantly...',
    '_embeddings' => [
        'title_vector' => [0.1, 0.2, 0.3, /* ... */],
        'content_vector' => [0.4, 0.5, 0.6, /* ... */],
    ],
]);

Batched embedding calls

When you pass many documents to merge(), Sigmie collects the texts from every doc and sends them to your embedding provider in batched requests — not one request per document.

What gets batched together

Sigmie groups texts by (api, dimensions, modality). Two docs whose title field uses the same provider and the same vector dimensions share a single request. Text and image inputs are never mixed in the same request, even when the same provider can handle both.

Chunk size

Each group is split into chunks of min(100, $provider->maxBatchSize()). The provider cap reflects what each backend accepts in one call — OpenAI and Jina allow 2048, Voyage 128, Cohere 96, Infinity 512.

$movies->merge($thirtyDocs);   // 1 request of 30
$movies->merge($twoHundredDocs); // 2 requests: 100 + 100

add() and replace() go through the same code path, so a single document is just a one-item batch.

What stays per-document

  • Docs that already carry an _embeddings block for a field (see Pre-computed embeddings) are excluded from the batch for that field.
  • After vectors come back, strategy formatting (concatenate / average / script-score) and score multipliers still run on each doc individually.

Tradeoffs

The win is fewer roundtrips and less rate-limit pressure when indexing in bulk. The cost: if a chunk fails, the whole merge() fails — Sigmie does not split-and-retry, because partial upserts produce inconsistent indexes. Keep merge batches at a size your provider can serve reliably.

Reranking

For higher-quality top-K results, rerank with a cross-encoder after retrieval:

use Sigmie\AI\APIs\InfinityRerankApi;
 
$sigmie->registerApi('my-rerank', new InfinityRerankApi(
    baseUrl: 'http://localhost:7998',
    model: 'cross-encoder/ms-marco-MiniLM-L-6-v2',
));
 
$response = $sigmie->newSearch('articles')
    ->properties($props)
    ->semantic()
    ->queryString('quantum computing applications')
    ->size(20)
    ->get();
 
$top5 = $response->rerank('my-rerank', ['content'], topK: 5);

See Retrieval and Agents for the full retrieval-then-generate pattern.

Empty queries

By default, an empty query string returns every document. To return nothing instead:

->noResultsOnEmptySearch()

Multilingual

Embeddings handle most languages automatically. A query in English finds documents in French if the embedding model supports both:

$sigmie->newSearch('documents')
    ->semantic()
    ->properties($props)
    ->queryString('machine learning')
    // matches "apprentissage automatique", "机器学习", etc.
    ->get();

Patterns

E-commerce search

$props = new NewProperties;
$props->name('name')->semantic(api: 'embeddings', dimensions: 512);
$props->text('description')->semantic(api: 'embeddings', dimensions: 512);
$props->category('category');
$props->price();
 
$sigmie->newSearch('products')
    ->properties($props)
    ->semantic()
    ->queryString('comfortable running shoes')
    ->filters('price<=100')
    ->get();

Content recommendation

$sigmie->newSearch('articles')
    ->semantic()
    ->disableKeywordSearch()
    ->properties($props)
    ->queryString('machine learning deep learning neural networks')
    ->size(5)
    ->get();

For “similar items” using existing documents as seeds, see Recommendations — it gives you RRF fusion and MMR diversification out of the box.

Troubleshooting

No results. Check that documents have embeddings — index a sample and inspect with $collection->get($id). If they don’t, verify your api: name matches the registered API.

Slow search. Drop to lower accuracy or smaller dimensions. Add filters to narrow the candidate pool before vector ranking.

Memory pressure. Lower accuracy and dimensions. High accuracy with 1536-dim vectors needs serious RAM at scale.

See also

  • Recommendations — “similar items” via vector retrieval with RRF and MMR.
  • Retrieval and Agents — combining retrieval, reranking, and generation.
  • Magic Tags — LLM-generated taxonomy tags backed by embeddings.
  • Docker — running local Infinity embeddings and reranker.