New Intelligent Developer Experiences for Compass, Atlas Charts, Relational Migrator, and Docs

Clark Gates-George, Shelby Carpenter, and Jon Lewis
September 26, 2023 | Updated: January 10, 2024

This post is also available in: Deutsch, Français, Español, Português.

Today, MongoDB announced a range of innovations in its developer data platform, creating new, intelligent developer experiences in familiar tools like MongoDB Compass, Atlas Charts, Relational Migrator, and MongoDB Documentation that radically simplify and accelerate how developers build modern applications.

These new experiences provide developers with guided and intelligent assistance for their development processes in:

  • MongoDB Compass: Where developers can use natural language to compose everything from simple queries to sophisticated, multi-stage aggregations.

  • MongoDB Relational Migrator: Where developers can convert SQL queries to MongoDB Query API syntax.

  • MongoDB Atlas Charts: Where developers can use natural language to generate basic data visualizations.

  • MongoDB Documentation: Where developers can ask questions to an intelligent chatbot, built on top of MongoDB Atlas and Atlas Vector Search, to enable lightning-fast information discovery and troubleshooting during software development.

Developer time is one of the most precious commodities in any organization, and with business and customer expectations continuing to rise, developers are under increasing pressure to deliver applications quickly. With more intelligent experiences across the MongoDB developer data platform, it is now simpler and easier than ever to build modern applications for virtually any use case.

Check out our AI resource page to learn more about building AI-powered apps with MongoDB.

Natural Language Queries in Compass

Building queries and aggregations is one of the most prominent developer use cases for Compass, MongoDB’s popular, downloadable GUI tool. Compass’ new, intelligent experience allows developers to use natural language to compose sophisticated aggregations to query, transform, and enrich data, reducing the complexity and learning curve to build queries into application code.

The new experience is being released in Public Preview in version 1.40.0 and will be rolled out incrementally to users starting today until the end of October.

To get started, make sure you have 1.40.0 downloaded on your machine and have access to the feature. Then you can navigate to the Documents tab and click on the Generate Query button in the query bar, which opens a second bar below the standard query bar where you can enter natural language prompts to generate the Query API syntax for you to execute against your data. Be sure to hit the “thumb’s up” or “thumb’s down” button to rate the helpfulness of the query generated.

SQL Query Conversion in Relational Migrator

Migrations are part of many developers’ journeys with MongoDB. Earlier this summer at MongoDB.Local NYC, we announced Relational Migrator to help teams with these projects, and we’re continuing to make it easier to modernize application code.

Many legacy systems have hundreds, if not thousands of SQL queries that must be modernized as part of any migration effort, and that can be a time-consuming, if not daunting task. Now in Private Preview, developers can use Relational Migrator to convert existing SQL queries and stored procedures into development-ready MongoDB Query API syntax.

With SQL query conversion, developers can leverage Relational Migrator to eliminate the manual effort of creating MongoDB queries at scale - speeding up migration projects.

SQL query conversion is currently available in Private Preview, and access can be requested directly from the latest version of Relational Migrator.

Natural Language Support in Atlas Charts

Atlas Charts is the best way for developers to visualize Atlas data. By offering an effortless and powerful solution for gaining data-driven insights, Charts empowers developers and the businesses they help scale.

What has always been easy is now becoming more intelligent too! Available in Private Preview, a new natural language mode allows developers to visualize their data through a simple language query, for example: “show me a comparison of annual revenue by country and product.” This is just the start. Later this year, natural language support will extend to more complex queries and chart types.

Sign up today to try out natural language support for building charts! Stay tuned for more updates from the team and check out our documentation to learn more about what’s supported by natural language during Private Preview!

Intelligent Chatbot for MongoDB Documentation

Documentation is critical to the developer experience, making it easier to discover product features and capabilities and troubleshoot common challenges during software development. MongoDB is now super-charging your experience with an intelligent chatbot that improves information discovery by surfacing and summarizing the most relevant documentation.

Built with MongoDB Atlas and Atlas Vector Search, the chatbot allows you to ask questions in natural language like “How do I get started with MongoDB Atlas?” or “How do I add a new IP address to the IP access list for my Atlas project?” and receive a response with reference articles, code examples, and other relevant information.

MongoDB will also be open-sourcing and providing educational materials about how we built the intelligent chatbot, making it that much easier for others in the community to use the power of MongoDB Atlas and Atlas Vector Search to create dynamic and educational experiences for their end users.

Data Privacy and Security

MongoDB is trusted by some of the world's most security-conscious organizations, who use the developer data platform’s robust data security and privacy controls to manage their most sensitive data assets. To maintain this trust, these new developer experiences will always be transparent about what data is accessed and used, allowing customers to make informed decisions within the boundaries of their unique security, privacy, and compliance concerns.

Get Started Today

With new, intelligent features that allow developers to interact with their data using natural language in Compass, Relational Migrator, and Charts, as well as an intelligent chatbot for MongoDB Documentation, it’s easier than ever to take advantage of the flexibility and scalability of MongoDB's document data model to build any class of application.

If you have feedback on these experiences, you can enter a suggestion in our user feedback portal.

← Previous

Introducing a Local Experience for Atlas, Atlas Search, and Atlas Vector Search with the Atlas CLI

This post is also available in: Deutsch , Français , Español , Português . Today, MongoDB is pleased to announce in Public Preview a new set of features for building software locally with MongoDB Atlas, giving developers greater flexibility and reducing operational overhead throughout the entire software development lifecycle. Developers can now develop locally with MongoDB Atlas deployments, including Atlas Search and Vector Search , using the Atlas CLI , empowering them to create full-text search or AI-powered applications no matter their preferred environment for building with MongoDB. Developers can use the Atlas CLI to set up, connect to, and automate common management tasks from early development through testing, staging, and production. For full-text search use cases, developers can now use the Atlas CLI to create and manage Atlas Search indexes regardless of whether they are working locally or in the cloud. Similarly, developers building applications powered by semantic search and generative AI on MongoDB can now use the Atlas CLI to create and manage local development instances with Vector Search indexes regardless of their development environment. Developer time is one of the most precious commodities in any organization building innovative new application experiences. But all too frequently, developers are burdened with managing repeatable tasks such as setting up development environments. They also often have to wrestle with the cognitive overhead of switching between different user experiences for local versus cloud development, distracting from delivering value. By giving developers the power of Atlas at their fingertips no matter their preferred development environment, MongoDB continues to expand the scope and capabilities of its developer data platform while placing a premium on developer experience. Create a Local Atlas Database Ready to create a local Atlas database, but don’t have the Atlas CLI yet? It’s easy to install with your favorite package manager. To install the Atlas CLI with Homebrew, use the following command: brew install mongodb-atlas In addition to installing via the Homebrew package manager, you can install the MongoDB Atlas CLI via Apt, Yum, Chocolatey, directly downloading the binary, or pulling the Docker image (learn more about our documentation ). You can also download it directly from the MongoDB Download Center . To create a local Atlas deployment with default settings in interactive mode, enter: atlas deployments setup --type local If you want to list your Atlas deployments enter: atlas deployments list If you’re authenticated to Atlas, you will see both your local and cloud Atlas deployments. If you aren’t authenticated to Atlas, you will only see your local deployments. Get Started with Local Atlas Search Building an application with a full-text search feature powered by Atlas Search? If you’re a developer who tends to build and prototype locally, you may be interested in using the Atlas CLI to work with Atlas Search in your local environment. To get started, first, connect to the local deployment on which you’d like to create a Search index: atlas deployments connect Next, you can use the MongoDB Shell to create your Search index. Below you’ll see an example of how to create an Atlas Search index: db.YOURCOLLECTION.createSearchIndex( "example-index", { mappings: { dynamic: true } } ) Then, if you want to run a query you can use the $search stage of an aggregation pipeline. You can learn more about managing Atlas Search indexes in our documentation . Get Started with Local Vector Search If you’re building an application with generative AI or semantic search and MongoDB Atlas, chances are you’ll be interested in our Atlas Vector Search offering. And now with the Atlas CLI, you can work with Vector Search in the cloud and your local environment. To get started with Vector Search locally you can use MongoDB Shell to create a Vector Search index. Notice that this is similar to the Atlas Search example above, except that in this case there is a vector embedding accounted for in search index creation. db.YOURCOLLECTION.createSearchIndex({ "mappings": { "dynamic": true, "fields": { "plot_embedding": { "type": "knnVector", "dimensions": 1536, "similarity": "euclidean" } } } } ) To learn more about running Vector Search queries visit our documentation . Additionally, if you're already familiar with handling your cloud Search indexes using the Atlas CLI, you'll appreciate a fresh set of interactive commands designed to help you efficiently manage Atlas Search and Vector Search indexes both locally and in the cloud: atlas deployments search indexes create From there you can move through an interactive flow that guides you through index creation. For detailed instructions visit our tutorial . Ready to Move to the Cloud? If you’re ready to create an Atlas database in the cloud, that is easy to do with the Atlas CLI. Simply use the following command: atlas deployments setup --type atlas From there, the setup wizard will guide you to: Register for an Atlas account or authenticate to an existing account Create a free MongoDB Atlas database Load sample data Add your IP address to the access list Create a database user and password Connect to the cluster using the MongoDB Shell ( mongosh ) so you can begin interacting with your data To learn more about the Atlas CLI, visit our documentation . And be sure to let us know what you think of the Atlas CLI in our user feedback portal . With the new local experience with the Atlas CLI, it’s easier than ever to work with your data on Atlas no matter your preferred development environment. Get started today with the Atlas CLI as the ultimate developer tool to manage MongoDB Atlas, including Atlas Search and Vector Search, throughout the entire software development lifecycle, from your local environment all the way to the cloud.

September 26, 2023
Next →

Search PDFs at Scale with MongoDB and Nomic

Data is only valuable if it’s accessible. For example, storing photos, audio files, or PDFs without the ability to extract information from them is like keeping junk in your basement, thinking you might need it someday. The problem is finding what you need to dig through your junk when the day comes. Until now, companies have followed a similar approach to unstructured data : store everything in data lakes for future use. But whether it’s junk in a basement or data in a data lake, the result is the same: accessibility is hard or impossible. However, the latest advancements in AI have disrupted this status quo. AI can effectively and efficiently compare similar objects by generating a vector representation or embedding a data object. This capability has revolutionized industries by enabling faster and more precise search, categorization, and recommendation systems than ever before. Whether it's being used to compare text, documents, images, or complex patterns in data, embeddings allow for nuanced interpretations and connections that were impossible with traditional methods. By taking advantage of AI, users can uncover insights and make unprecedented speed and accuracy decisions. A particularly interesting use case is PDF search, since every company in the world deals with PDFs in one way or another. While PDFs allow portability across platforms and operating systems, most PDF readers only allow for basic exact-match queries. PDF search powered by MongoDB and Nomic Enter MongoDB and Nomic: MongoDB Atlas Vector Search with Nomic Embed equips organizations with a powerful and affordable AI-powered search solution for large PDF collections. A machine learning company specializing in explainable and accessible AI, Nomic Embed is the company’s flagship text embedding model with out-of-the-box features suitable for scalable PDF search. Its features include: Long context: Nomic Embed breaks new ground by supporting a long context length of 8192 tokens, exceeding the standard 2048. This extended context makes the model ideal for real-world applications that involve processing large PDFs and documents. High throughput: While achieving top performance on the MTEB embedding benchmark, Nomic Embed is smaller than similarly performing models. At only 137 million parameters and 548MB, Nomic Embed enables high-throughput embedding generation for data-heavy workflows or streaming applications. Flexible storage: Nomic Embed provides adjustable embedding size via Matryoshka representation learning. Users can freely choose to store the first 64, 128, 256, or 512 embedding dimensions out of the full 768, depending on their project requirements. Smaller embedding sizes come at a minimal performance loss while providing lower storage costs and faster computing benefits. To put Nomic Embed’s abilities in context, consider a company that processes a high volume of PDFs—say 100,000 documents per month—with an average length of 20 pages each. To improve database retrieval speed, these documents can be partitioned into smaller chunks, such as 2 pages per chunk (see Figure 1 below). Assuming a full page typically contains around 500 words, each document chunk would consist of approximately 1000 words. Figure 1: PDF chunking, embedding creation with Nomic, and storage into MongoDB Embedding models process words as numerical tokens where a general rule of thumb is 3/4 word = 1 token. One embedding is more than sufficient to represent a document chunk in this case, as 4/3 * 1000 tokens fit nicely in Nomic Embed’s long context window. A PDF search application for this company would require 100,000 PDFs x 10 chunks = 1,000,000 embeddings. Benchmarked on Nomic’s AWS Sagemaker real-time inference offering on a single GPU ml.g5.xlarge instance, the total runtime is under 4 hours for a total of $15.60 per month. A similar performing embedding model such as OpenAI’s text-embedding-3-small costs $26.66 per month to generate the same number of embeddings. Once the embeddings are stored in MongoDB Atlas, it’s possible to create an Atlas Vector Search index to unlock their potential. Building a PDF search application at this point becomes straightforward. The query text is vectorized, and the embedding is fed to Atlas Vector Search to retrieve similar vectors. The result is a list of the most semantically similar sections of the PDF relevant to the original text. This is a significant leap forward compared to a simple “ctrl-f” search, as it captures meaning rather than just keyword matches. This process can be further improved by implementing a retrieval-augmented generation (RAG) pipeline, combining Atlas Vector Search and a large language model (LLMs). As shown in Figure 2, this approach allows users to ask questions in natural language about the content of the PDF. The relevant documents are then fed to the LLM as context, and the AI is able to provide structured answers by leveraging knowledge about the data. Figure 2: Retrieval Augmented Generation flow with Nomic In a nutshell, Nomic and MongoDB provide the building blocks for advanced RAG applications, equipping developers with a cost-effective and integrated toolset. Seamless integration, supercharged search: Nomic Embeddings in MongoDB Atlas MongoDB Atlas seamlessly ingests Nomic embeddings with its flexible document storage format. Depending on the application, embeddings and additional metadata can be neatly stored together or separately in MongoDB collections. MongoDB Atlas and Nomic Embed are both available as AWS Marketplace offerings for same-VPC deployments. MongoDB Atlas Stream Processing is a perfect fit for Nomic Embed’s high throughput capabilities. Incoming data streams are robustly processed and can be combined with MongoDB Database Triggers to generate embeddings for immediate downstream use. Given Nomic Embed’s lightweight nature and offline capabilities (via private or local deployments from open source), embeddings can be produced and ingested into MongoDB at extremely rapid transfer rates. MongoDB Atlas Vector Search delivers a fast and accessible method to leverage Nomic embeddings for semantic search . MongoDB Atlas Vector Search lets you combine these fast vector search queries with traditional database queries on various metadata, providing a flexible and powerful analytics tool for data insights, user recommendations, and more. Industry use cases PDFs are ubiquitous. In one way or another, every company in the world needs to extract and analyze PDF content to make business decisions or comply with regulations. Let’s have a look at some industry use cases: Financial services The financial services industry is constantly bombarded with essential updates, including market data, financial statements, and regulatory changes. Some of this information such as financial statements, annual reports, and regulatory filings, resides in PDF format. Efficient and reliable navigation through these documents is crucial for gaining a competitive edge in investment decision-making. For example, investors scrutinize key financial metrics such as revenue growth, profit margins, and cash flow trends extracted from income statements, balance sheets, and cash flow statements. They use this information to compare them between companies, gauging their strategic direction, risks, and competitive positioning before investing. However, accessing and extracting data from these PDFs can be a time-consuming challenge, hindering agility in the fast-paced financial landscape. Here, semantic search for financial PDFs offers a dramatic improvement in information discovery. By leveraging semantic search technology, which interprets the intent and contextual meaning behind a search query, FSI professionals can significantly enhance their ability to find relevant information. This applies equally to the broader financial industry, including areas like market analysis, performance evaluation, and many more. Retail In the retail industry, the challenge of processing hundreds of thousands of invoices from numerous suppliers annually is a common scenario. Most invoices are in PDF format, and the challenge arises from the combination of invoice volume and the variability in layouts and languages from one supplier to another. This makes manual processing impractical and error-prone. The question becomes: how can retailers automate this end-to-end process efficiently and accurately? The answer lies in solutions that utilize advanced technologies like AI and PDF search capabilities. By leveraging these solutions, retailers can automatically scan invoices, extract relevant data, and validate it against purchase orders and received goods. Moreover, these solutions offer the flexibility to adapt to different invoice layouts without the need for templates, ensuring scalability and efficiency gains. With increased automation rates and improved accuracy levels, retailers can shift focus from low-value manual tasks to more strategic initiatives, accelerating their digital transformation journey and unlocking significant cost savings along the way. Manufacturing & motion There are vast amounts of unstructured data contained in PDFs across the Manufacturing and Automotive industries, from machine instruction booklets to production or maintenance guidelines, Six Sigma best practices, production results, and team lead annotations. All this valuable data must be shared, read, and stored manually, introducing significant friction when it comes to leveraging its full potential. With MongoDB Atlas Vector Search, manufacturing companies have the opportunity to completely revive this data and make real use of it in their day-to-day operations, all while reducing the time spent managing these manuals and having everything ready to be accessed. It is as simple as vectorizing the documents, uploading them to MongoDB Atlas, and connecting a RAG-enabled application to this data source. With this, operators in a manufacturing plant can describe a problem to a smart interface and ask how to troubleshoot it. The interface will retrieve the specific parts of the manual that show how to address the issue. Moreover, it can also retrieve notes from previous operators, team leaders, or previous troubleshooting efforts, providing a very rich context and accelerating the problem-solving process. PDF RAG-enabled applications in manufacturing open up a wide range of operational improvements that directly benefit the company's bottom line. PDF search at scale In today’s data-driven world, extracting insights from unstructured data like PDFs is challenging. Traditional search methods fall short, but advancements in AI like Nomic Embed have revolutionized PDF search. By leveraging MongoDB with Nomic Embed, organizations gain a powerful and cost-effective AI-powered solution for large PDF collections. Nomic Embed’s extensive context, high throughput capabilities, and MongoDB’s seamless integration and powerful analytics enable efficient and reliable PDF search applications. This translates to enhanced data accessibility, faster decision-making, and improved operational efficiency. Don't waste time struggling with traditional PDF search! Apply for an innovation workshop to discuss what’s possible with our industry experts. If you would like to discover more about MongoDB and GenAI: Building a RAG LLM with Nomic Embed and MongoDB From Relational Databases to AI: An Insurance Data Modernization Journey

April 30, 2024