Pinterest has managed to slash its AI costs by an impressive 90% while simultaneously boosting accuracy by 30% by overhauling its approach to using frontier models for image recommendations. With 620 million monthly active users, the stakes are high, and the costs of calling a frontier model for every image recommendation are equally significant. CTO Matt Madrigal and his team tackled this financial drain by dismantling the vision layer of the Qwen3-VL model and replacing it with proprietary embeddings, marking a strategic shift in the company’s AI deployment.
## How Pinterest Customized Qwen for Visual Discovery
Pinterest has long been a proponent of open-source models for its visual search and discovery features, utilizing technologies like Google’s BERT and OpenAI’s CLIP. The company has developed its own customized version, Pin CLIP, integrating proprietary visual embeddings and image metadata to enhance its capabilities. The recent overhaul of Qwen3-VL for their conversational shopping assistant, Navigator 1, involves a significant reconfiguration of the model’s vision encoder layer. By incorporating their own multimodal embeddings, Pinterest can precompute metadata surrounding pins and images offline, allowing for regular retraining to enhance user personalization.
The use of open-source models, particularly those with open Apache licenses, is a strategic move for Pinterest. It allows the company to tailor these models extensively to its unique use cases. According to Madrigal, without these customized embeddings, the system would suffer from a latency “20 times worse” at runtime, due to the need to encode each image individually. This bottleneck would be unsustainable for a platform with such a massive user base, making their approach not only cost-effective but also crucial for maintaining performance standards.
## How a Taste Graph Captures Evolving Interests
Beyond customizing AI models, Pinterest has developed a “taste graph” to better understand and predict user preferences, guiding them from inspiration to purchase. Unlike traditional search engines, Pinterest focuses on users in the discovery phase, promoting what Madrigal refers to as “lateral exploration” to convert discovery into actionable intent, such as ad clicks or purchases.
The taste graph combines a graph structure with representational learning to maintain a dynamic profile of user interests. These user embeddings are continuously updated, capturing the nuances of individual tastes as they evolve. This system is distinct from a social graph, focusing instead on personal preferences and potential inspirations. For example, users with a penchant for mid-century modern designs will receive recommendations aligned with that style, enhancing the likelihood of engagement and conversion.
## Real Implications for Founders, Engineers, and the Industry
Pinterest’s approach illustrates a critical lesson for engineers and founders: the value of leveraging and customizing open-source models to fit unique business needs. As AI continues to proliferate across industries, the ability to refine and adapt existing models could become a competitive differentiator. For startups and smaller companies, the financial savings and performance gains demonstrated by Pinterest could serve as a blueprint for managing costs without sacrificing quality.
Moreover, the concept of a taste graph underscores the importance of understanding user behavior at a granular level. By focusing on preference-driven data rather than just clicks, companies can create more meaningful and effective user interactions, ultimately driving higher engagement and conversion rates.
## What Happens Next
As Pinterest continues to refine its AI capabilities, the focus will likely remain on enhancing personalization and efficiency. For industry stakeholders, this serves as a reminder of the importance of adaptability and the potential benefits of customizing open-source technologies. Engineers and developers should consider how similar strategies might be applied to their own projects, potentially unlocking new opportunities for cost savings and performance improvements.
