How to Use MongoDB in Python with MongoEngine

Introduction

MongoDB is a free and open-source NoSQL database program. It stores data in collections of documents where a document is like a row in the traditional relational database systems. It does not have a fixed schema for storing data in a collection; you store data in key-value pairs like a Python dictionary. The features like flexible data models and support for horizontal scaling allow you to make changes or scale the database as the requirements change.

MongoEngine is an Object Document Mapper (ODM) that allows your Python application to interact with MongoDB databases. It offers a declarative API where you can interact with documents in the database using Python classes and objects. This abstraction layer makes it easier for developers to interact with MongoDB databases, helps reduce the development time, and makes database logic less error-prone.

This article explains the installation and configuration of the MongoEngine ODM, the steps to define a document schema, perform Create, Read, Update and Delete (CRUD) operations, and query filtered data from the database. It also goes over the basics of document reference, migration, and meta options.

Prerequisites

• Deploy an Ubuntu 20.04 instance at Vultr.
• Create a non-root user with sudo privileges.
• Log in to your instance as the non-root user.
• Install MongoDB database server on the instance.

Set Up the Environment

Install the MongoEngine library.

$ pip install mongoengine

Set up a new project directory.

$ mkdir ~/mongoengine_demo

Switch to the project directory.

$ cd ~/mongoengine_demo

Connect to the Database

You use the connect() function in the MongoEngine module to establish a connection with the MongoDB server. You must pass the values such as hostname, port, database, and more as parameters.

Using a text editor, create a new Python file named app.py.

$ nano app.py

Add the following contents to the file and save the file using Ctrl + X then Enter.

from mongoengine import *

client = connect('mongoengine_demo')

The above code establishes a connection with the MongoDB server using default host 127.0.0.1 and port 27017. The string passed as a parameter refers to the name of the database. It tries to create a new database if it does not exist, using the name passed to the connect() function.

The following example demonstrates how to connect to a MongoDB server with non-default values.

connect('database_name', username='username', password='password', authentication_source='admin')

The above statement uses keyword attributes specifying the database credentials in the connect() function. Refer to the pymongo.MongoClient API reference for a complete list of the accepted parameters.

You can also use the alias parameter for establishing connections with multiple databases or database servers. Refer to the Multiple Databases section in the MongoEngine documentation for more information.

Define Document Schema

The documents stored in a MongoDB database do not have any fixed schema. However, defining a document schema ensures the data structure and validation, making the database logic less error-prone.

The MongoEngine ODM allows you to create a fixed or dynamic document schema. You can inherit the Document or DynamicDocument class from the MongoEngine module to create new document classes and define each field using the field objects, refer to Field section in the MongoEngine documentation to find out all available field types.

Edit the Python file created in the previous section.

$ nano app.py

Add the following contents to the file and save the file using Ctrl + X then Enter.

class User(Document):
    name = StringField(required=True)
    email = EmailField(required=True)
    age = IntField(required=True)

    def __repr__(self):
        return f'<User name="{self.name}">'

The above code creates a document class named User inherited from the Document class in the MongoEngine module. It maps the Python class to a collection in the database. By default, It uses the Python class name converted into snake case as the collection name. This document class uses a fixed schema where an object can only contain the name, email, and age.

Perform CRUD Operations

The MongoEngine maps each document class to a collection in the database. You create a new instance of the document class to create a new document in the collection. This section explains how to perform CRUD operations using the document class created in the previous section.

Enter the Python Console.

$ python

Import the required modules.

>>> from app import User

The above command imports the User class from the Python file you created in the previous section.

Create a new entry.

>>> user1 = User(name='Example User', email=f'user@example.com', age=21)
>>> user1.save()

The above command creates a new object from the User class and calls the save() method, which initiates the collection and creates a new document in the database.

Read the first entry.

>>> User.objects.first()

The above command returns a dictionary with the values of the first document stored in the user collection.

Update an entry.

>>> user1.age = 22
>>> user1.save()

The above command changes the value of the age attribute in the user1 object and calls the save() method, which updates the document in the database.

Verify the changes.

>>> User.objects.first().age

Delete an entry.

>>> user1.delete()

Verify the changes.

>>> User.objects

Exit the Python console.

>>> exit()

Query Filtered Data

The document classes have an objects attribute that allows accessing the objects stored in the collection. The objects attribute is a QuerySetManager that accepts conditions and returns a QuerySet object containing the filtered document objects. This section explains the basics of querying filtered data using the document class.

Enter the Python Console.

$ python

Import the required modules.

>>> from app import User

Populate the database.

>>> user_objects = [User(name=f'Person {i}', email=f'person{i}@example.com', age=i+18) for i in range(10)]
>>> User.objects.insert(user_objects)

The above commands create a list of 10 different User objects with enumerated values and insert it into the database.

Query documents using a single condition.

>>> User.objects(age__gt=20)

The above command returns a list of documents with ages more than 20.

Output

[<User name="Person 3">, <User name="Person 4">, <User name="Person 5">, <User name="Person 6">, <User name="Person 7">, <User name="Person 8">, <User name="Person 9">]

Query documents using multiple conditions.

>>> User.objects(age__gt=20, age__lt=25)

The above command returns a list of documents with ages more than 20 and less than 25.

Output

[<User name="Person 3">, <User name="Person 4">, <User name="Person 5">, <User name="Person 6">]

Fetch a single document.

>>> User.objects(age__=19).first()

The above command returns a single document object with an age of 19. The first() method returns a single document object instead of a list containing the document object.

Output

<User name="Person 1">

Exit the Python console.

>>> exit()

Refer to the Querying the database in the MongoEngine documentation to find all available filtering options in the QuerySetManager.

Document Reference

The MongoEngine ODM allows linking to other documents in the document schema. It enables you to create a relation between the documents; like in the traditional relational database systems, they store the primary key of the linked row as a foreign key.

You use the ReferenceField field object in the document schema to link to other documents. The MongoEngine ODM also supports the combination of the ListField and ReferenceField field objects to form many-to-one relations.

This section explains how to use the ReferenceField field object in document classes to link to different documents.

Edit the Python file.

$ nano app.py

Add the following contents to the file above the User document class and save the file using Ctrl + X then Enter.

class Video(Document):
    title = StringField(required=True)

    def __repr__(self):
        return f'<Video title="{self.title}">'

class Course(Document):
    name = StringField(required=True)
    price = IntField(required=True)
    videos = ListField(ReferenceField(Video))

    def __repr__(self):
        return f'<Course name="{self.name}">'

The above code creates Video and Course Python classes inherited from the Document class creating two new collections in the database. The Course document class uses the ReferenceField and ListField field objects to store a list of video object references in the videos attribute.

Enter the Python Console.

$ python

Import the required modules.

>>> from app import Course, Video

The above command imports the Course and Video classes from the app.py file.

Create new entries in the video collection.

>>> video1 = Video(title='Example Video 1').save()
>>> video2 = Video(title='Example Video 2').save()

The above commands create two new instances of the Video document class that initiates and creates two new documents in the database.

Create a new entry in the course collection.

>>> course1 = Course(name='Example Course 1', price=100, videos=[video1, video2]).save()

The above command creates a new instance of the Course document class that initializes and creates a new document in the database. It links the document with two other documents in the video collection.

Verify the changes.

>>> Course.objects.first().videos
>>> Course.objects.first().videos[0].title
>>> Course.objects.first().videos[1].title

Exit the Python console.

>>> exit()

Document Migration

The NoSQL database's flexible nature allows you to make document schema changes easily. This section explains how to make structural changes in the document schema.

Edit the Python file

$ nano app.py

Add the following contents to the file and save the file using Ctrl + X then Enter.

class User(Document):
    name = StringField(required=True)
    email = EmailField(required=True)
    age = IntField(required=True)
    enrolled_courses = ListField(ReferenceField(Course))

    def __repr__(self):
        return f'<User name="{self.name}">'

The above code modification adds a new attribute named enrolled_courses in the user collection. The combination of ListField and ReferenceField field objects allows a user document to refer to multiple course documents.

Enter the Python Console.

$ python

Import the required modules.

>>> from app import User, Course

The above command imports the User and Course classes from the app.py file.

Fetch the document objects.

>>> user1 = User.objects.first()
>>> course1 = Course.objects.first()

The above commands fetch the first document objects from the user and the course collection.

Update the user document object.

>>> user1.enrolled_courses = [course1]
>>> user1.save()

The above commands set the value of the enrolled_courses attribute as a Python list containing the Course objects for document reference.

Verify the changes.

>>> User.objects.first().enrolled_courses
>>> User.objects.first().enrolled_courses[0].name

Exit the Python console.

>>> exit()

The changes in the document schema do not affect the existing documents. If you want to apply the changes to all the documents, you must use the update_many() method on the collection. For more information, refer to the Document Migration section in the MongoEngine documentation.

Document Meta Options

The meta dictionary in document classes allows you to add metadata to the collection, such as collection name, list of document indexes, default ordering and inheritance options, sharding keys, and so on. This section explains the usage of the meta dictionary in the document class.

Edit the Python file

$ nano app.py

Add the following contents to the file below the User document class and save the file using Ctrl + X then Enter.

class MetaExample(Document):
    age = IntField()

    meta = {
        'collection': 'example_collection',
        'indexes': ['age']
    }

    def __repr__(self):
        return f'<MetaExample age={self.age}>'

The above code creates a new Python class named MetaExample inherited from the Document class. It creates a new collection named example_collection and adds the age field in the collection indexes.

Indexes are a special data structure that stores a subset of the data stored in the database, making the data easier to transverse programmatically. Without indexes, the server must perform a full collection scan. If an appropriate index exists for the query, the server can limit the number of documents it must scan.

Enter the Python Console.

$ python

Import the required modules.

>>> from app import client, MetaExample

The above command imports the client object and MetaExample class from the app.py file.

Initialize the collection.

>>> MetaExample(age=999).save()

Verify the custom collection name.

>>> client.get_database('mongoengine_demo').list_collection_names()

Verify the collection indexes.

>>> MetaExample.list_indexes()

Exit the Python console.

>>> exit()

Refer to the Document collections section in the MongoEngine documentation to find the complete list of available meta options.

Conclusion

You installed the MongoEngine library, established a connection with the MongoDB database server, defined the document schema, performed CRUD operations, and queried filtered data from the database. You also explored the basics of document reference, migration, and meta options.

More Information

• MongoEngine Documentation
• PyMongo Documentation
• MongoDB Documentation