To search data, you must organize it into indexes<\/strong>. Indexes store documents (sets of fields with key-value pairs) and optimize them. Optimization is possible because each field has a specific type. You can specify field types. Otherwise, OpenSearch can try to determine the type automatically.<\/p>\n\n\n\n Another form of optimization is splitting the index into several shards<\/strong>. Each shard contains a subset of the documents inside the index. When you search for data, queries run across different shards in parallel if each shard is located on a different node. The size of shards should be around 10-30 GB for workloads requiring low search latency and 30-50 GB for write-heavy workloads such as storing logs.<\/p>\n\n\n\n OpenSearch instances are called nodes<\/strong>. OpenSearch can operate as a single-node or multi-node cluster. When creating a multi-node cluster, the number of nodes, the node types, and their hardware depend on your use case.<\/p>\n\n\n\n The types of nodes<\/strong> are:<\/p>\n\n\n\n A node can have multiple types. Each node is a master-eligible, data, ingest, and coordinating node by default.<\/p>\n\n\n\n During the creation of an AWS OpenSearch cluster, you can customize Data Nodes and Dedicate Master Nodes.<\/p>\n\n\n\n For Data Nodes, you can specify the number of nodes, the instance type, the volume type, and the volume size.<\/p>\n\n\n\n Dedicated Master Nodes are nodes that do not contain data and are dedicated to cluster management. You can choose to not have any Dedicate Master Node in development or test environments. Since they do not contain data, you must specify only the number of nodes and the instance type.<\/p>\n\n\n\n On AWS, you can deploy an OpenSearch cluster in two different ways:<\/p>\n\n\n\n The classic approach is the Serverfull one in which you have to choose how many nodes you want to create, specifying the node types, sizing, and other properties.<\/p>\n\n\n\n Let\u2019s look at the AWS Console and go through the Amazon OpenSearch service. The first step we will do is create a domain, in other words, an OpenSearch cluster.<\/p>\n\n\n\n You will have to choose between two options: Easy create<\/strong> or Standard create.<\/strong><\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n\n\n\n We strongly recommend choosing the Standard Create option since it is more flexible, while you could use the Easy create <\/strong>one just for short demos or POCs.<\/p>\n\n\n\n Going deep into the Standard creation, you can select from two different templates: Production or dev\/test. We suggest skipping this choice and going through the other parameters.<\/p>\n\n\n\n The most important selection is the kind of redundancy of the cluster:<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n Pay attention to this option since it will drastically change your pricing.<\/p>\n\n\n\n In fact, the option Domain with standby <\/strong>will create a cluster with minimum of 3 data nodes, and you can increment them only by multiples of 3. <\/p>\n\n\n\n Meanwhile, with the Domain without standby, <\/strong>you can customize the number of nodes, but none of them will be reserved as standby nodes.<\/p>\n\n\n\n Which is the best-fitting solution for your use case? As a rule of thumb, choose the Domain with a standby option for mission-critical workloads. For other scenarios, just choose the other option. <\/p>\n\n\n\n AWS provides OpenSearch Serverless, an on-demand, auto-scaling configuration for Amazon OpenSearch Service. OpenSearch Serverless provides collections: groups of indexes with a common use case. Collections are the serverless counterpart of clusters, but collections don\u2019t require manual provisioning. OpenSearch Serverless allows two types of collections: Time series collections<\/strong> and Search collections<\/strong>.<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n The main difference is that in search collections, all data is stored in hot storage to ensure fast query response times; instead, time series collections use a combination of hot and warm caches to optimize query response times for more frequently accessed data.<\/p>\n\n\n\n Also, you can index by custom ID only in search collections.<\/p>\n\n\n\n OpenSearch Serveless compute capacity is measured in OpenSearch Compute Units (OCUs). Each OCU is a combination of 6 GiB of memory, corresponding virtual CPU, and data transfer to Amazon S3.<\/p>\n\n\n\n The first Serverless collection instantiates a total of four OCUs (2 used for ingesting, primary, and standby; 2 used for searching, and one active replica for high availability). Subsequent collections can share these OCUs. OpenSearch scales OCUs based on the indexing and searching workloads. These four initial OCUs are always active, even when there’s no indexing or search activity, so you are billed for at least four OCUs. You can set a maximum number of OCUs to contain costs. You are also billed for storage retained in Amazon S3.<\/p>\n\n\n\n Ok, now we know everything about the creation of an OpenSearch cluster, but how can we use it? Through Queries<\/strong>.<\/p>\n\n\n\n OpenSearch provides a search language called query domain-specific language (DSL) which provides a JSON interface. OpenSearch also supports SQL to write queries rather than use DSL.<\/p>\n\n\n\n There are two main types of queries: leaf queries<\/strong> and compound queries<\/strong>.<\/p>\n\n\n\n Leaf queries search for a specified value in a certain field or fields. Leaf queries can be further categorized into different sub-types.<\/p>\n\n\n\n Full-text queries are used to search text documents. There are different types of full-text queries.<\/p>\n\n\n\n You can match a specific value in a specific field or search for the values in a list of fields. You can assign a weight to each field to boost its value in the result score. For instance, while searching for a book, finding the value in the \u201ctitle\u201d field could have a bigger impact than finding it in the \u201cabstract\u201d field.<\/p>\n\n\n\n A type of query allows you to search for different terms in a field, and the last term of the input string will be used as a prefix to have documents that contain either any of these terms or terms starting with the prefix.<\/p>\n\n\n\n Some full-text queries support a \u201cfuzziness\u201d parameter useful in case of inputs written with missing characters or characters whose position is exchanged.<\/p>\n\n\n\n Others support a \u201cslop\u201d parameter that controls how words can be misordered and still be considered a match.<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n Term-level queries are used to search documents for an exact specified term.<\/p>\n\n\n\n You can search for exact or multiple terms in a field. When searching for multiple terms, term-level queries also allow you to specify the minimum number of matches required. You can search for an ID, a value contained in a range, or terms containing a specific prefix. Fuzzy queries search for terms similar to the search term using the Levenshtein distance. Term-level queries offer the possibility to search using wildcard or Lucene regular expression. You can also search for documents containing a specific field instead of a value.<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n XY queries search for documents that contain geometries using xy_point and xy_shape fields. xy_point fields support points. xy_shape fields support points, lines, circles, and polygons.<\/p>\n\n\n\n You can search for documents whose points or shapes intersect the provided shape or those whose shapes do not intersect, are contained, or contain the provided shape.<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n Geographic queries are used to search documents containing geospatial geometries. Geographic queries support points and shapes like lines, circles, and polygons. <\/p>\n\n\n\n You can search for documents whose geopoint values are within a bounding box or a polygon. Geoshape queries return documents that contain geopoints or geoshapes that intersect the provided shape, or documents that contain geoshapes that do not intersect, are contained, or contain the provided shape. You can also query for documents with geopoints within a specified distance from a provided geopoint.<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n Joining queries can be used to query a nested object as an independent document or retrieve parent or child documents linked through a field of \u201cjoin\u201d type.<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n Compound queries wrap multiple leaf queries to combine their results or modify their behavior. You can join multiple query clauses with boolean logic or assign a higher score to documents that match multiple clauses. Compound queries also offer the ability to create a function to recalculate returned documents’ scores. Lastly, you can combine a \u201cpositive\u201d query with a \u201cnegative\u201d query, documents found through the positive query will have an increased score, while documents found through the negative query will have a decreased score. This is useful in case of synonyms that you want to remove from your results.<\/p>\n\n\n\n There are two main purposes for using the OpenSearch service: one is to store and query\/analyze application and infrastructure logs & metrics<\/strong>, and the other is to build a search engine<\/strong>.<\/p>\n\n\n\n This is one of the main uses of OpenSearch. Thanks to the fast query response time in a lot of data, it allows very fast searching for specific application or infrastructure logs. <\/p>\n\n\n\n The main purpose of this solution is to have a centralized log storage where you can find all the applications and infrastructure logs and build near real-time dashboards that visualize different metrics to understand your application’s healthiness.<\/p>\n\n\n\n Let’s imagine a web application composed of a Single Page Application front-end and a back-end hosted on AWS using serverless services like Amazon API Gateway and AWS Lambda. What you can do is send all the logs to AWS OpenSearch, structuring your log payload with at least this information:<\/p>\n\n\n\n And then add different attributes based on what you are logging for. For example, we can imagine having at least two logs for every request: one when the request starts that will add the event source as the additional attributes and one when the request ends, with an attribute that defines the response status code. If an error happens, we can imagine logging the same payload with an error status code.<\/p>\n\n\n\n Just with this information, we can build a simple Dashboard that shows us the following:<\/p>\n\n\n\n And we can filter all the logs for a specific time range or even for the request id showing us the entire log of the single request.<\/p>\n\n\n\n Another main purpose of using OpenSearch is to build a Search Engine. Thanks to all the different types of queries that the engine supports, you can simply create a search bar that uses fuzziness to search all the data that look like what your user has typed. <\/p>\n\n\n\n You can even use the match_phrase_prefix function to autocomplete your user input based on the data that you have in the database or use the suggest function to correct your user input.<\/p>\n\n\n\n In this article, we have given a general view of what Opensearch is, its fundamentals, and how to leverage AWS services to manage a cluster of this kind. Nowadays, so many workloads require such data sources, so you must know it!<\/p>\n\n\n\n Many topics have not been taken into consideration, such as how to authenticate and authorize to an OpenSearch cluster, how to leverage Amazon S3 as warm and cold data storage for the cluster, enable Cognito as Identity Provider, or how to manage Disaster Recovery for business-critical applications.<\/p>\n\n\n\n If you are interested in these or other aspects of OpenSearch implementations, please leave a comment below! <\/p>\n\n\n\n More content like this? Choose your topic<\/a> and subscribe to our newsletter!<\/em><\/p>\n\n\n\n
Provisioning types<\/h2>\n\n\n\n
\n
Serverfull<\/h3>\n\n\n\n
![\"create](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2023\/08\/create-domanin-opensearch-1024x707.png\")
<\/figure>\n\n\n\n![\"cluster](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2023\/08\/image1-1024x671.png\")
<\/figure><\/div>\n\n\nServerless<\/h3>\n\n\n\n
![\"collection](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2023\/08\/image5-1024x284.png\")
<\/figure><\/div>\n\n\nQuery types<\/h2>\n\n\n\n
Leaf Queries<\/h3>\n\n\n\n
Full-text queries<\/h4>\n\n\n\n
![\"\"](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2023\/08\/image3.png\")
<\/figure><\/div>\n\n\nTerm-level queries<\/h4>\n\n\n\n
![\"\"](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2023\/08\/image6.png\")
<\/figure><\/div>\n\n\nXY queries<\/h4>\n\n\n\n
![\"\"](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2023\/08\/image4.png\")
<\/figure><\/div>\n\n\nGeographic queries<\/h4>\n\n\n\n
![\"\"](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2023\/08\/image2.png\")
<\/figure><\/div>\n\n\nJoining queries<\/h4>\n\n\n\n
![\"\"](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2023\/08\/image8.png\")
<\/figure><\/div>\n\n\nCompound queries<\/h3>\n\n\n\n
Use cases<\/h2>\n\n\n\n
Logs & Metrics<\/h3>\n\n\n\n
\n
\n
Search Engine<\/h3>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
![\"subscribe](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2023\/08\/Screenshot-2023-08-01-at-15.54.03.png\")
<\/figure><\/div>\n\n\n