Gemini Embedding 2 connects text, images, audio and video

Google has opened Gemini Embedding 2 to public preview. The model can convert text, images, video, audio and documents into a shared representation, allowing an application to find relationships between them. It is available through the Gemini API and Vertex AI.

This matters because traditional search often separates each file type. A search engine may locate words in a document, but it does not always understand that an image, a sentence and an audio clip are about the same topic. Embeddings, numerical representations of a piece of content's meaning, let you compare those elements by what they mean, not just by the words they contain.

One model for multiple formats

Gemini Embedding 2 places different types of content in a shared space. This allows an application to search for an image using a written description, connect a video to a question or group audio recordings with documents about the same subject.

The model supports:

• Text with a context of up to 8,192 tokens, equivalent to several thousand words.
• Up to 6 images per request, in PNG and JPEG formats.
• Videos up to 120 seconds long, in MP4 and MOV.
• Audio processed directly, without having to transcribe it first.
• PDF files of up to 6 pages.

It also understands combined inputs. You can send an image together with text in the same request so it can capture the relationship between them instead of analyzing each element separately. For example, a company could search its catalog for product images that also match a specific written description.

Fewer pieces to build an application

One practical consequence is that teams can avoid several steps and specialized models. In a multimodal search application, they would no longer need one tool to transcribe audio, another to analyze images and another to index text before trying to combine the results.

Google points to uses such as:

• Semantic search, which finds related content even when it does not use the same words.
• RAG, a technique that retrieves relevant information before generating a response.
• Opinion and sentiment analysis across different formats.
• Classification and clustering of large data collections.
• Management of files that combine documents, images, audio and video.

The model also lets you choose the size of the final representation. Its default setting uses 3,072 dimensions, but this can be reduced to 1,536 or 768. That lets you trade some quality for lower storage and search costs. Google recommends 3,072, 1,536 and 768 dimensions when you want the highest possible quality.

What changes for you

For end users, the change may show up in tools that can search across an entire library, not just through text. A query could find a page in a PDF, a scene in a video and a voice recording because they all express a similar idea.

Gemini Embedding 2 is in Public Preview, so it is still offered as a test version for developers. Google says it outperforms leading models on text, image and video tasks, and adds strong audio capabilities, although the announcement does not publish the specific figures from those comparisons here.

The important questions will be how it performs in real-world cases and how much it costs to use at scale. If it maintains a consistent level across formats, it could simplify the development of search engines and assistants that have so far required several different tools to work together.