Introduction
MongoDB is a popular, open-source, NoSQL database management system that is designed to store and manage large volumes of unstructured or semi-structured data. MongoDB is known for its flexibility, scalability, and ease of use, making it a preferred choice for many modern applications, particularly those that deal with large datasets and require high-performance data access.
Data is stored in the form of collections and documents and they look very similar to JSON objects with key:value pairs.
Mongo DB setup
Setting up MongoDB involves several options depending on your development and deployment requirements.
We can install MongoDB locally on our computer, develop and deploy that, OR
We can use a cloud database that is already deployed and it is much more easier for us to manage. One such cloud service is Mongo DB Atlas.
In this blog, we will learn how to install Mongo DB locally and work with it. Later we will also learn how to set up Mongo DB Atlas.
Install Mongo DB Community Edition (Check the Mongo DB compass option during installation) and Mongo Shell.
Collections & Documents
Data is stored in the form of collections and documents in NoSQL databases. A database can have any number of collections for different types of data.
For example, we have Users collection to store user data, a Blog Posts collection to store blog data etc.
Each collection stores data in the form of documents. WKT the Blog Posts collection contains data about different blogs. The information of each blog is nothing but a document.
Each document stores data using a data structure called BSON (Binary JSON). Each document has a unique id that is used to identify that document and is created automatically by the database (_id) every time we create a new document.
Mongo DB Compass
MongoDB Compass is a graphical user interface (GUI) and visualization tool provided by MongoDB for interacting with and managing MongoDB databases. It is a desktop application that makes it easier for developers, database administrators, and data analysts to work with MongoDB databases by offering a more user-friendly and visual way to explore and manipulate data.
When you open the application, it
The input text area expects a connection string. Connection string is a special Mongo DB URL used to connect to a Mongo DB cluster like the Mongo DB Atlas. Since we are working with Mongo DB locally, we will leave the input box empty and just click on the connect button. This will connect the application to the local Mongo DB service. Now the UI will look something like this.
Now we can add a new custom database and add collections and required documents to it.
We can delete, update, duplicate, and filter the documents of any collection.
Mongo DB Shell
The MongoDB shell, often referred to simply as the "mongo shell," is a command-line interface (CLI) tool provided by MongoDB to interact with MongoDB databases. It allows developers, database administrators, and system administrators to perform various database operations, including querying, inserting, updating, and managing data, directly from the command line.
We can either use the Mongo DB Shell integrated with Mongo DB Compass or we can use the Windows terminal and type mongosh to start the terminal.
Querying The Database
We have a list of all commands that we can use to query the database here. However, we will look into some of the basic and important commands.
Adding new documents:
//Adding a single document db.books.insertOne({ title: "The Color of Magic", author: "Terry Pratchett", pages: 300, rating: 7, genres: ["fantasy", "magic"] }) //We will get an acknowledgement object as a response { acknowledged: true, insertedId: ObjectId('622fdsfsdfsfs') }If we are referring to a nonexisting collection, the database will create a new collection with the given name and insert the document into it.
We can also insert multiple documents at a time by using
insertMany()method and pass an array of documents.db.books.insertMany([doc1, doc2,....., docN])Finding/Fetching Documents:
db.books.findOne({_id: "622fdkjsdsd"}) //returns a single document db.books.find() //Returns the first 20 documents. "it" to display more db.books.find().limit(3) //Returns only the first 3 documents //Find all books whose author is 'Terry Pratchett' AND rating=7 db.books.find({author: "Terry Pratchett", rating: 7}) //Adding filters //Fetching only required properties -> Can be specified as 2nd argument db.books.find({author: "Terry Pratchett"}, {title: 1, rating: 1}) //We can leave the filters empty: {} //We can chain different methods after find() methods db.books.find().count() //No of documents in books collection db.books.find({author: "Terry Pratchett"}).count() //With filter db.books.find().sort({title: 1}) //Sorts the fetched docs based on title db.books.find().sort({title: -1}) //Sorts in descending orderNested Documents
Nested documents in MongoDB refer to the practice of defining a document inside another document.
{ "title": "The Way of Kings", "genres": ["fantasy", "sci-fi"], "rating": 9, "author": "Brandon Sanderson", "_id": Objectid("ai5eg8H9Pk12"), "reviews": [ { "name": "Great Read!", "body": "Lorem ispum...." }, { "name": "So sol guess", "body": "Lorem ispum...." }, { "name": "My fav every book", "body": "Lorem ispum...." } ] }In this example, the
reviewsfield is a nested document that contains its own set of fields. This nesting allows you to represent the review information within the same document as the book, making it easy to retrieve and work with related data.Advantages of using nested documents in MongoDB:
Data Encapsulation: Nested documents encapsulate related data, making it easier to understand and maintain the structure of your data.
Atomic Updates: MongoDB allows atomic updates within a single document. You can modify fields within the nested document without affecting other parts of the document.
Query Efficiency: Queries that involve nested data can be more efficient because all the related information is stored together in one document, reducing the need for complex joins.
Schema Flexibility: MongoDB's flexible schema allows you to add or remove fields within nested documents without affecting other documents in the same collection.
Performance: For certain use cases, nested documents can lead to improved read and write performance, as they reduce the need for multiple database operations.
Simplicity: Nested documents simplify data modeling by avoiding the need to create separate collections and manage relationships using references, as is common in relational databases.
We use a . operator to refer to the nested document. When we are using . operator, we have to enclose the selector in double quotes.
//Querying nested documents
db.books.find({"reviews.name": "Luigi"}) //Fetching books revied by Luigi
Operators & Complex Queries
In MongoDB, operators are special keywords or symbols used in queries and update operations to perform specific actions or comparisons of data. An operator is identified by a $ in its prefix.
Comparison Operators:
$eq: Matches values that are equal to a specified value.$ne: Matches values that are not equal to a specified value.$gt: Matches values that are greater than a specified value.$lt: Matches values that are less than a specified value.$gte: Matches values that are greater than or equal to a specified value.$lte: Matches values that are less than or equal to a specified value.$in: Matches any of the values specified in an array.$nin: Matches values that are not in the specified array.
Logical Operators:
$and: Joins query clauses with a logical AND and returns documents that match all the conditions.$or: Joins query clauses with a logical OR and returns documents that match at least one of the conditions.$not: Inverts the effect of a query expression and returns documents that do not match the specified condition.$nor: Joins query clauses with a logical NOR and returns documents that do not match any of the conditions.
Let us look at some example queries using operators.
db.books.find({ rating: {$gt: 7}}) //Fetch books with rating > 9
db.books.find({ rating: {$gt: 7}, author: "Patrick Rothfuss"}) //rating > 9 and author is Patrick Rothfuss
db.books.find({$or: [{rating: 7}, {rating: 9}]}) //rating = 7 or 9
//Books with page number < 300 OR page number > 400
db.books.find({$or: [{pages: {$1t: 300}}, {pages: {$gt: 400}}]})
db.books.find({ rating: {$in: [[7,8,9]}}) //rating is either 7,8 or 9
db.books.find({$or: [{rating: 7}, {rating: 8}, {rating: 9}]})//Equivalent to the above query
You can try writing queries using other operators as well. Use chat-gpt if needed.
Array Operators:
$all: Matches documents where an array field contains all specified elements.$elemMatch: Matches documents where at least one array element meets the specified condition.$size: Matches documents where the array field's size matches a specified value.
db.books.find({ genres: "fantasy" }) //Documents which has fantasy as one of its genres
db.books.find({ genres: ["fantasy"] }) //Documents which has fantasy as the ONLY genres
db.books.find({ genres: ["fantasy", "sci-fi"] }) //fantasy and sci-fi
//Documents which has 'fantasy', 'sci-fi' as genres. The genres can contain other values as well
db.books.find({ genres: {$all: ["fantasy", "sci-fi"] } })
Deleting Documents:
db.books.deleteOne({_id: ObjectId("622fmslkdfsd7")})//Deletes doc with given id db.books.deleteOne({author: "Terry Pratchett"})//Deletes the FIRST doc that is encoountered with author=Terry Pratchett db.books.deleteMany({author: "Terry Pratchett"}) //Deletes ALL docs with author=Terry PratchettUpdating Documents:
//Updates rating and pages of document with given id db.books.updateOne({_id: ObjectId("622b7460")}, {$set: rating: 8, pages: 200}}) //Updates author of all documents with written by Terry Patchett db.books .updateMany({author: "Terry Pratchett"}, {$set: {author: "Terry"}}) //Increments,Decrements the no of pages by 2 db.books.updateOne({_id: ObjectId("622b746cd2d41")}, {$inc: {pages: 2}}) db.books.updateOne({_id: ObjectId("622b746cd2d41")}, {$inc: {pages: -2}})Some array modification operators
$push: Appends a value to an array field.$pull: Eliminates all instances of a value from an array field.$each: Modifies array fields to append multiple values simultaneously.
//Removes 'fantasy' from genres array
db.books.updateOne({_id: ObjectId("622b7U6d41")}, {$pull: {genres: "fantasy"}})
//Appends 'fantasy' to genres array
db.books.updateOne({_id: ObjectId("622b7U6d41")}, {$push: {genres: "fantasy"}})
//Pushes all elements in each array to the genres array
db.books.updateOne({_id: ObjectId("622b7c86f1")}, {$push: {genres: {$each: ["sci-fi", "action"]}}})
Mongo DB Drivers
So far we have seen Mongo DB interactions in their purest form without using any specific programming environment like Python, Node JS, Ruby etc. and used only Mongo DB Shell and Compass to interact with the database. But most of the time, we will be interacting with Mongo DB from an application code like Node JS. To facilitate that, we have to use Mongo DB Drivers. Each programming language has its driver. We will be using the Node.js driver for the rest of the blog.
First of all, If you want to follow along you should know the basics of how node applications work and how to make a simple express app. If you are completely new to Node, I would highly recommend you go through my blogs on Node.js and expresss.js.
Set up the Mongo DB driver by reading this document.
Connecting to the Database
We will create a new file db.js to contain the database connection code.
const { MongoClient } = require('mongodb')
let dbConnection
module.exports = {
connectToDb: (cb) => {
MongoClient.connect('mongodb://localhost:27017/bookstore')
.then(client => {
dbConnection = client.db()
return cb()
})
.catch(err => {
console.log(err)
return cb(err)
})
},
getDb: () => dbConnection
}
The above code exports an object with two functions:
connectToDb(cb): This function is used to establish a connection to the MongoDB database. It takes a callback functioncbas a parameter, which will be called once the connection is established. Inside this function:It uses
MongoClient.connectto connect to the MongoDB server running locally on port 27017 and the "bookstore" database.When the connection is successfully established, it assigns the database connection to the
dbConnectionvariable and calls the provided callback function (cb) without any error as the argument.If there is an error during the connection process, it logs the error to the console and also calls the callback function with the error as an argument.
getDb(): This function is used to retrieve the database connection. It simply returns thedbConnectionvariable, which should contain the connected database object after callingconnectToDb.
Creating an express app(app.js) to interact with the database:
const express = require('express')
const { getDb, connectToDb } = require('./db')
const app = express()
let db
connectToDb((err) => {
if(!err){
app.listen('3000', () => {
console.log('app listening on port 3000')
})
db = getDb()
}
})
app.get('/books', (req, res) => {
res.json({mssg: "welcome to the api"})
})
Here we are importing the methods from db.js. The first job we need to do is to connect to the database, hence we call the connectToDb() method to connect to the database. If there is no error, we can start listening to requests.
For now, we are just returning dummy data as a response to requests on the route '/books'.
Cursors and Fetching Data
The syntax we used earlier in the Mongo DB Shell to fetch data was db.books.find(), which fetched the first 20 documents from the database. But in the Node application, we do it a little differently.
app.get('/books', (req, res) => {
let books = []
db.collection('books')
.find()
.sort({author: 1})
.forEach(book => books.push(book))
.then(() => {
res.status(200).json(books)
})
.catch(() => {
res.status(500).json({error: 'Could not fetch the documents'})
})
})
Before diving deep into the code, let us understand what cursors are in Mongo DB, this will help us better understand the code.
When we execute db.collection('books').find(), the find() method does not return all the documents in the database like before, it returns something called the cursor. It is nothing but an object that points to the set of documents that we want to fetch. If we have not added any filters, it points to the whole collection of documents else it points to the subset of documents that satisfy the conditions mentioned in the filter.
Now the cursor can be used to iterate through the documents in the "books" collection. However, at this point, the query hasn't been executed, and no documents have been retrieved from the database. It's important to note that this is a query builder, and the actual data retrieval happens when you start iterating through the cursor using forEach() or use methods like toArray().
toArray()is a method that can be chained with thefind()method on a cursor. It's used to retrieve all the documents from the cursor and convert them into an array.db.collection('books').find().toArray((err, documents) => { if (err) { console.error('Error:', err); return; } // 'documents' is an array containing all the documents console.log(documents); });forEach()is another method that can be chained with thefind()method on a cursor. It's used to iterate through the documents one by one and apply a function to each document.let books = [] db.collection('books').find().forEach(book => books.push(book))
Now the code that we wrote earlier is pretty self-explanatory. When there is a request from the route '/books', we fetch documents from the 'bookStore' collection using db.collection('books').find() and storing it in the books[] array. Then we return a JSON object of the books[] array.
Now let us look at how to fetch a single document.
app.get('/books/:id', (req, res) => {
if (ObjectId.isValid(req.params.id)) {
db.collection('books')
.findOne({_id: new ObjectId(req.params.id)})
.then(doc => {
res.status(200).json(doc)
})
.catch(err => {
res.status(500).json({error: 'Could not fetch the document'})
})
}
else {
res.status(500).json({error: 'Could not fetch the document'})
}
})
First, understand what route parameters are.
Route Parameters: Route parameters are flexible components within a route that can vary in value. For instance, in the URL "localhost:3000/blogs/:id," the term "id" functions as a route parameter, marked by the use of ":" in its prefix. The value of "id" can differ, adapting to different blog entries, such as "localhost:3000/blogs/1," "localhost:3000/blogs/2," and so on. This mechanism enables dynamic handling of distinct resources under a unified route structure. We can get the route parameter of a request using req.params.parameterName.
While fetching a single document, will be using the id of the document as the route parameter. Now let us look at the working of the code.
ObjectId.isValid(req.params.id): To check whether theidparameter provided in the URL is a valid MongoDB ObjectId.db.collection('books').findOne(id): If the Object Id is valid, we fetch a document with the given id. If fetch was successful, we return the fetched document as a JSON object with status code 200, else we return an error object with status code 500.
Using Postman Application
Postman allows us to simulate requests through API and shows the response that we get back from the request. The main reason we use Postman instead of the browser is that to make POST, DELETE and UPDATE requests, we would require a frontend JavaScript to do it on the browser. Instead, we can use Postman to simulate those requests very easily.
Download Postman Application from here and signup.
When we send a GET request to the URL, we get back a JSON object as a response. We can save this request in a new collection or an existing collection.
Now we are all set to send different types of requests to the server from Postman.
Handling POST Request
app.use(express.json())
app.post('/books', (req, res) => {
const book = req.body
db.collection('books')
.insertOne(book)
.then(result => {
res.status(201).json(result)
})
.catch(err => {
res.status(5@0).json({err: ‘Could not create a new document’
})
});
We cannot use req.body directly to extract the data from the POST request. We need a middleware for that: app.use(express.json()). It is used to parse incoming JSON payloads from HTTP requests.
Sending POST request from Postman.
Handling DELETE Request
app.delete('/books/:id', (req, res) => {
if (ObjectId.isValid(req.params.id)) {
db.collection('books')
.deleteOne({_id: ObjectId(req.params.id)})
.then(result => {
res.status(200).json(result)
})
.catch(err => {
res.status(500).json({error: 'Could not delete the document'})
})
}else {
res.status(500).json({error: 'Document Not Found'})
}
});
Sending a DELETE request from Postman.
Handling PATCH Requests
The PATCH request is used to update an existing document in a collection. But we learned earlier in Node.js that we use a PUT request to update a document right? Let us understand how a PATCH request is different from a PUT request.
The
PATCHmethod is used to apply partial modifications to a resource. It means that you send a request to the server with a specific set of changes you want to apply to the resource, and the server applies those changes to the resource.The
PUTmethod is used to update a resource or create it if it doesn't exist. When you make aPUTrequest, you send the entire representation of the resource to the server, and the server replaces the existing resource with the new representation you provide.
app.patch('/books/:id', (req, res) => {
if (ObjectId.isValid(req.params.id)) {
const updates = req.body
db.collection('books')
.updateOne({_id: ObjectId(req.params.id)}, {$set: updates})
.then(result => {
res.status(200).json(result)
})
.catch(err => {
res.status(500).json({error: 'Could not update the document'})
})
}else {
res.status(500).json({error: 'Document Not Found'})
}
});
Sending a PATCH request from Postman.
Pagination
Pagination in MongoDB involves splitting a large result set of documents into smaller, manageable subsets or "pages" to improve the user experience when displaying data in a web application. In the example we have been discussing so far, say we had a thousand books (documents) in the 'books' collection, we wouldn't get all the books at once but rather get a certain amount of them at a time say 20. This is called pagination.
If we want to see more books, the user can request them by clicking the next button. That is what we are trying to achieve in this section. To implement this, we use something called query parameters.
Query Parameters: Query parameters are a way to pass information to a web server as part of a URL (Uniform Resource Locator). They are typically used in HTTP requests, such as GET requests, to provide additional data or instructions to the server. Query parameters are included in the URL after a question mark (?) and are separated by ampersands (&) if there are multiple parameters.
Here's the basic structure of a URL with query parameters:http://localhost:3000/books?p=1
For the above URL, the query parameter is p. Handling these query parameters is simple in an express application. Syntax: req.query.parameter-name
We would require two additional methods to implement pagination
skip()Method: Theskip()method is used to specify the number of documents to skip from the beginning of the result set.limit()Method: Thelimit()method is used to specify the maximum number of documents to return in the result set.
app.get('/books', (req, res) => {
let books = []
const page = req.query.p || 0 //If value of p is not specified then p=0
const booksPerPage = 20
db.collection('books')
.find()
.sort({author: 1})
.skip(page * booksPerPage) //Skip the first page*books-per-page pages
.limit(booksPerPage) //After skipping the above pages, consider only booksPerPage pages
.forEach(book => books.push(book))
.then(() => {
res.status(200).json(books)
})
.catch(() => {
res.status(500).json({error: 'Could not fetch the documents'})
})
})
To implement this in the front end we can increment page number every time we click the next button and fetch the books again belonging to that page.
Mongo DB Atlas
Till now, we have made a node-express API that communicates with Mongo DB locally on our computer. When it comes to a production application, your app won't be communicating with your local Mongo DB server, instead it will be communicating with the Mongo DB server online. One way to do this is to use Mongo DB Atlas, which is a database as a service and allows us to set up a free cloud database and connect with our application very easily.
You can set up Mongo DB Atlas here.
Login/Signup into Mongo DB
Build a new cluster (Stick to default settings).
Create a database under the
collectionstab. Create desired collections in the database.Go to the
database accesstab and create a new database user. You will be accessing the database from your express app using this username and password.Go back to the
clusterstab and get your connection string. We will use this in our express app. Replace <username> and <password> in the connection string with the actual username and password that you created while signing up.
Now the final step, replace the address of the local Mongo DB server with the connection string in db.js.
MongoClient.connect('mongodb://localhost:27017/bookstore')
||
\/
const uri = 'mongodb+srv: //<username>:<password>@cluster@.del96.
mongodb.net/myFirstDatabase?retryWrites=true&w=majority'
MongoClient.connect(uri)
Conclusion
In conclusion, this blog has provided a comprehensive overview of MongoDB, covering essential aspects from setting up MongoDB, working with collections and documents, to querying the database using various methods like MongoDB Compass and the MongoDB Shell. We delved into the power of nested documents, explored operators for complex queries, and learned about different MongoDB drivers for connecting to the database.
We also ventured into practical applications with discussions on handling HTTP requests using the Postman application, including POST, DELETE, and PATCH requests. Additionally, we addressed the important topic of pagination for efficient data retrieval.
Lastly, we touched upon MongoDB Atlas, a cloud-based solution that facilitates database management and scaling for your applications. MongoDB Atlas opens up new possibilities for deploying and managing MongoDB databases with ease.
Keep exploring, experimenting, and honing your MongoDB skills to unlock the full potential of this database technology.