Azure AI Search

Azure AI Search (formerly Azure Cognitive Search) is Microsoft's managed search service and the default retrieval layer behind Azure OpenAI's "On Your Data" and AI Foundry RAG workflows. It supports full-text (BM25), vector, semantic reranking, and hybrid search in a single index, with built-in skillsets that call other Azure AI services (OCR, language, vision) during ingestion.

Key Features:

• Hybrid Retrieval: BM25 + vector in one query; reciprocal rank fusion (RRF) merges the two result sets.
• Semantic Ranker: Microsoft's cross-encoder reranker rescores top-N results using a Bing-derived language model — typically +10–20% NDCG over vector alone.
• Integrated Vectorization: Chunk, embed, and index documents directly from Blob Storage, OneLake, ADLS, or SharePoint — no separate pipeline.
• Skillsets & Indexers: Declarative enrichment graph that runs OCR (Document Intelligence), language detection, key-phrase extraction, image captioning, and custom Web API / Azure Function skills during ingestion.
• Security Filters: Index-time document-level ACLs with Entra ID group membership passed at query time.
• Scoring Profiles: Boost by recency, popularity, field matches, or geo distance without custom code.
• Vector Quantization: Scalar and binary quantization cut vector storage 4×–32× with minimal recall loss.

Index Schema Essentials:

• Fields: Each has a type (Edm.String, Collection(Edm.Single) for vectors, etc.) and attributes (searchable, filterable, sortable, facetable, retrievable, key).
• Vector Fields: Dimension and vector search profile specifying algorithm (HNSW or exhaustive-KNN) and compression.
• Semantic Configuration: Tells the semantic ranker which fields are title, content, and keywords.
• Analyzers: Per-field language analyzers (standard.lucene, en.lucene, custom) control tokenization and stemming.

Examples

1. Create an Index with Text + Vector Fields

from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
    SearchIndex, SearchField, SearchFieldDataType, SimpleField, SearchableField,
    VectorSearch, HnswAlgorithmConfiguration, VectorSearchProfile,
    SemanticConfiguration, SemanticField, SemanticPrioritizedFields, SemanticSearch,
)
from azure.core.credentials import AzureKeyCredential

client = SearchIndexClient("https://myco-search.search.windows.net", AzureKeyCredential(""))

index = SearchIndex(
    name="hr-policies",
    fields=[
        SimpleField(name="id", type=SearchFieldDataType.String, key=True),
        SearchableField(name="title", type=SearchFieldDataType.String),
        SearchableField(name="content", type=SearchFieldDataType.String, analyzer_name="en.microsoft"),
        SearchField(
            name="content_vector",
            type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
            searchable=True,
            vector_search_dimensions=1536,
            vector_search_profile_name="hnsw-default",
        ),
    ],
    vector_search=VectorSearch(
        algorithms=[HnswAlgorithmConfiguration(name="hnsw")],
        profiles=[VectorSearchProfile(name="hnsw-default", algorithm_configuration_name="hnsw")],
    ),
    semantic_search=SemanticSearch(configurations=[
        SemanticConfiguration(
            name="default",
            prioritized_fields=SemanticPrioritizedFields(
                title_field=SemanticField(field_name="title"),
                content_fields=[SemanticField(field_name="content")],
            ),
        )
    ]),
)
client.create_or_update_index(index)
  

2. Hybrid Search with Semantic Reranking

from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizedQuery
from openai import AzureOpenAI

aoai = AzureOpenAI(azure_endpoint="...", api_key="...", api_version="2024-10-21")

def embed(text: str) -> list[float]:
    return aoai.embeddings.create(model="embedding-3-large-prod", input=[text]).data[0].embedding

search = SearchClient("https://myco-search.search.windows.net", "hr-policies", AzureKeyCredential(""))

results = search.search(
    search_text="parental leave for new parents",      # BM25 side
    vector_queries=[VectorizedQuery(                   # vector side
        vector=embed("parental leave for new parents"),
        k_nearest_neighbors=50,
        fields="content_vector",
    )],
    query_type="semantic",                             # turns on semantic reranker
    semantic_configuration_name="default",
    top=5,
    select=["id", "title", "content"],
)

for r in results:
    print(f"{r['@search.reranker_score']:.3f}  {r['title']}")
  

3. Integrated Vectorization — Index from Blob Storage, No Pipeline Code

Point an indexer at a data source, attach a skillset that chunks + embeds, and Azure AI Search handles ingestion end-to-end.

{
  "name": "hr-skillset",
  "skills": [
    {
      "@odata.type": "#Microsoft.Skills.Text.SplitSkill",
      "textSplitMode": "pages",
      "maximumPageLength": 2000,
      "pageOverlapLength": 300,
      "inputs":  [{"name": "text", "source": "/document/content"}],
      "outputs": [{"name": "textItems", "targetName": "chunks"}]
    },
    {
      "@odata.type": "#Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill",
      "resourceUri": "https://myco.openai.azure.com",
      "deploymentId": "embedding-3-large-prod",
      "modelName":    "text-embedding-3-large",
      "context": "/document/chunks/*",
      "inputs":  [{"name": "text", "source": "/document/chunks/*"}],
      "outputs": [{"name": "embedding", "targetName": "chunk_vector"}]
    }
  ]
}
  

4. Filtered Query for Document-Level Security

user_groups = ["hr-admin", "all-employees"]
filter_expr = " or ".join(f"acl_groups/any(g: g eq '{g}')" for g in user_groups)

results = search.search(
    search_text="termination policy",
    filter=filter_expr,
    top=10,
)
  

Common Use Cases:

• RAG grounding for Azure OpenAI "On Your Data" and AI Foundry agents.
• Enterprise search across SharePoint, OneDrive, Confluence, and ADLS.
• Product catalog search with vector + facets + scoring profiles.
• Conversational support knowledge bases with semantic reranking.
• Multi-tenant SaaS search with per-tenant security filters.