Introduction

So far, we've explored some of the earliest embedding models, including word2vec, doc2vec, and BERT. In this lesson, we're going to take it up a notch and learn about far more powerful embedding models that we can call via API.

Meet OpenAI's Embedding API. OpenAI originally launched their Embedding API in January 2022 and have since rolled out a series of models that offer state-of-the-art performance and highly competitive pricing.

Unlike the local models we've used in the previous lessons, such as doc2vec and BERT, OpenAI's embedding models can be accessed remotely via API. In many ways, this is a massive advantage, as it allows us to leverage state-of-the-art model without having to worry about resource constraints imposed by our local machine and the difficulties of setting up complex machine learning infrastructure. However, it also means that we have to be mindful of the costs associated with using the API.

Below, we'll explore how OpenAI's API provides a cost-effective yet powerful option for generating embeddings. Even better, we'll put OpenAI's embedding models to the test by building out a semantic search pipeline for querying paragraphs across Paul Graham's essay collection.