OpenSearch 3.6 debuts with new observability stack

OpenSearch has released version 3.6, the project's first long-term support release.

The update adds a new OpenSearch Observability Stack, Application Performance Monitoring, and a set of search and agent tools for developers and operations teams.

One of the main additions is OpenSearch Launchpad, a guided tool that helps users build search applications from sample documents and user inputs. Through a conversational workflow, it can provision a local setup, choose retrieval approaches, configure models, and generate a working user interface.

Version 3.6 also introduces OpenSearch Relevance Agent as an experimental feature. It is designed to automate search relevance tuning by analysing user behaviour, generating hypotheses, and evaluating changes through a natural-language interface in OpenSearch Dashboards.

The release also includes an experimental multi-agent orchestration platform for Relevance Agent and other specialised agents. It adds a unified agent registration API that combines several manual setup steps into a single API call, as well as a new conversational_v2 agent type with standardised inputs for text, documents, images, and message history.

Observability Tools

A major part of the release focuses on observability. The new OpenSearch Observability Stack combines OpenTelemetry Collector, Data Prepper, OpenSearch, Prometheus, and OpenSearch Dashboards in a preconfigured deployment that can be launched through Docker Compose or an installer.

The stack is intended for monitoring microservices, web applications, and AI agents. It includes distributed tracing, service maps, log analytics, Prometheus-compatible metrics, and tools for tracking large language model calls, execution graphs, and token usage.

OpenSearch 3.6 also adds Application Performance Monitoring, or APM, for distributed applications. It provides service topology maps, RED metrics covering rate, errors, and duration, and links from service metrics to related traces and logs inside an Observability workspace in Dashboards.

Users can group services by attributes such as SDK language or team, filter by error or fault-rate thresholds, and inspect time-series charts for requests, latency, and errors. The release also adds Agent Traces, aimed at monitoring generative AI applications through OpenTelemetry-based instrumentation.

A Python SDK supports providers including OpenAI, Anthropic, Amazon Bedrock, LangChain, and LlamaIndex, along with frameworks such as Strands Agents, LangGraph, and CrewAI. In Dashboards, the Agent Traces plugin displays execution flows in graph and timeline views, alongside token usage and span-level details.

Search Changes

On the search side, the release includes a series of upgrades to vector search and relevance tools. Version 3.6 introduces 1-bit scalar quantisation across both the Faiss and Lucene engines, with 32x compression in each case.

According to the project, Faiss is now the default method and delivers 24% better recall and 15% lower latency than existing binary methods. Lucene gains 1-bit scalar quantisation for the first time, enabling approximate and exact k-nearest neighbour search on quantised vectors.

Other changes are intended to reduce latency and storage use. These include a 40% reduction in latency for quantised index searches through Faiss optimisations, metadata compression using Zstandard, and prefetch functionality for approximate nearest neighbour and exact search workloads that can halve search latency in memory-constrained settings.

Search Relevance Workbench has also been updated. Users can now run experiments against multiple data sources, organise work with name and description fields, create query sets in the interface, and use additional metrics including Recall@K, Mean Reciprocal Rank, and Discounted Cumulative Gain.

Query Analysis

The release also expands query analysis and debugging functions. A new filter setting for top query insights allows visibility to be restricted by user identity or shared team roles, while administrators retain full access.

There is also a recommendation engine for top queries. It can inspect query structures and index metadata, detect anti-patterns, and propose fixes with confidence scores and estimated effects on latency, CPU, memory, and correctness.

For short-lived queries, version 3.6 adds a finished-queries cache layer to the Live Queries API. The update also introduces remote blob storage support for top N query data through Amazon S3 repositories and adds visual analytics including percentile statistics, pie charts, tables, line charts, and heatmaps.

Query language is another area of change. The project has added new and updated PPL functions, including external query libraries for third-party tools, result highlighting, auto-extract mode for spath, recursive graph traversal through graphlookup, and query cancellation and timeout controls.

In announcing the release, OpenSearch said: "The latest version of OpenSearch includes an array of agent-powered tools to help you build, deploy, and deliver results faster."