Improving Search with OpenAI's Embedding API
tooling
project
3
Project Definition: Paul Graham Essay Search
The other day, I was reading a new Paul Graham essay and thought to myself, "It would be helpful to have a tool that could retrieve relevant snippets from his vast collection of essays." So, why not build one? We can use OpenAI's embedding API to cheaply embed each paragraph from his archive of essays, and then use those embeddings to retrieve the most relevant paragraphs for a given query. A classic semantic search problem!
🔎 If you aren't familiar with him, Paul Graham is a well-known computer scientist, entrepreneur, and essayist. He was a co-founder of Y Combinator and has written hundreds of essays on a wide variety of topics, from startups to philosophy to programming languages. You can find his essays at paulgraham.com/articles.html.
Of course, since there's no public dataset of all his essays, we'll need to scrape and preprocess them ourselves into a corpus of paragraph documents.