Hunterdigest

Migrations are Django’s way of propagating changes you make to your models(adding a field, deleting a model, etc.) into your database schema. They’redesigned to be mostly automatic, but you’ll need to know when to makemigrations, when to run them, and the common problems you might run into.

Apr 17, 2016 The built-in publish functionality in Visual Studio is also going to enable the ability to run migrations as a deployment step - rather than at application startup (that should ship in RC2). And then of course you can also use the migrations commands to get a SQL script if you have a more formalized workflow. The enable-migrations command creates a Migrations folder in the ContosoUniversity project, and it puts in that folder a Configuration.cs file that you can edit to configure Migrations. Oct 28, 2020 If you're more comfortable working inside Visual Studio or have experience with EF6 migrations, you can also use the Package Manager Console tools. Create your first migration You're now ready to add your first migration! Instruct EF Core to create a migration named InitialCreate. The code-based migration provides more control on the migration and allows you to configure additional things such as setting a default value of a column, configure a computed column etc. In order to use code-based migration, you need to execute the following commands in the Package Manager Console in Visual Studio. Create Visual Studio Solution(Windows only) Enable this setting to generate Visual Studio Solution files for your Project, so you can build your final executable in Visual Studio. Create Xcode Project (Mac Only) Enable this setting to generate an Xcode project so you can build your final application bundle in Xcode.

The Commands¶

There are several commands which you will use to interact with migrationsand Django’s handling of database schema:

• migrate, which is responsible for applying and unapplyingmigrations.
• makemigrations, which is responsible for creating new migrationsbased on the changes you have made to your models.
• sqlmigrate, which displays the SQL statements for a migration.
• showmigrations, which lists a project’s migrations and theirstatus.

You should think of migrations as a version control system for your databaseschema. makemigrations is responsible for packaging up your model changesinto individual migration files - analogous to commits - and migrate isresponsible for applying those to your database.

The migration files for each app live in a “migrations” directory insideof that app, and are designed to be committed to, and distributed as partof, its codebase. You should be making them once on your development machineand then running the same migrations on your colleagues’ machines, yourstaging machines, and eventually your production machines.

Note

It is possible to override the name of the package which contains themigrations on a per-app basis by modifying the MIGRATION_MODULESsetting.

Migrations will run the same way on the same dataset and produce consistentresults, meaning that what you see in development and staging is, under thesame circumstances, exactly what will happen in production.

Django will make migrations for any change to your models or fields - evenoptions that don’t affect the database - as the only way it can reconstructa field correctly is to have all the changes in the history, and you mightneed those options in some data migrations later on (for example, if you’veset custom validators).

Backend Support¶

Migrations are supported on all backends that Django ships with, as wellas any third-party backends if they have programmed in support for schemaalteration (done via the SchemaEditor class).

However, some databases are more capable than others when it comes toschema migrations; some of the caveats are covered below.

PostgreSQL¶

PostgreSQL is the most capable of all the databases here in terms of schemasupport.

The only caveat is that prior to PostgreSQL 11, adding columns with defaultvalues causes a full rewrite of the table, for a time proportional to its size.For this reason, it’s recommended you always create new columns withnull=True, as this way they will be added immediately.

MySQL¶

MySQL lacks support for transactions around schema alteration operations,meaning that if a migration fails to apply you will have to manually unpickthe changes in order to try again (it’s impossible to roll back to anearlier point).

In addition, MySQL will fully rewrite tables for almost every schema operationand generally takes a time proportional to the number of rows in the table toadd or remove columns. On slower hardware this can be worse than a minute permillion rows - adding a few columns to a table with just a few million rowscould lock your site up for over ten minutes.

Finally, MySQL has relatively small limits on name lengths for columns, tablesand indexes, as well as a limit on the combined size of all columns an indexcovers. This means that indexes that are possible on other backends willfail to be created under MySQL.

SQLite¶

SQLite has very little built-in schema alteration support, and so Djangoattempts to emulate it by:

• Creating a new table with the new schema
• Copying the data across
• Dropping the old table
• Renaming the new table to match the original name

This process generally works well, but it can be slow and occasionallybuggy. It is not recommended that you run and migrate SQLite in aproduction environment unless you are very aware of the risks andits limitations; the support Django ships with is designed to allowdevelopers to use SQLite on their local machines to develop less complexDjango projects without the need for a full database.

Workflow¶

Django can create migrations for you. Make changes to your models - say, add afield and remove a model - and then run makemigrations:

Your models will be scanned and compared to the versions currentlycontained in your migration files, and then a new set of migrationswill be written out. Make sure to read the output to see whatmakemigrations thinks you have changed - it’s not perfect, and forcomplex changes it might not be detecting what you expect.

Once you have your new migration files, you should apply them to yourdatabase to make sure they work as expected:

Once the migration is applied, commit the migration and the models changeto your version control system as a single commit - that way, when otherdevelopers (or your production servers) check out the code, they’llget both the changes to your models and the accompanying migration at thesame time.

If you want to give the migration(s) a meaningful name instead of a generatedone, you can use the makemigrations--name option:

Version control¶

Because migrations are stored in version control, you’ll occasionallycome across situations where you and another developer have both committeda migration to the same app at the same time, resulting in two migrationswith the same number.

Don’t worry - the numbers are just there for developers’ reference, Djangojust cares that each migration has a different name. Migrations specify whichother migrations they depend on - including earlier migrations in the sameapp - in the file, so it’s possible to detect when there’s two new migrationsfor the same app that aren’t ordered.

When this happens, Django will prompt you and give you some options. If itthinks it’s safe enough, it will offer to automatically linearize the twomigrations for you. If not, you’ll have to go in and modify the migrationsyourself - don’t worry, this isn’t difficult, and is explained more inMigration files below.

Transactions¶

On databases that support DDL transactions (SQLite and PostgreSQL), allmigration operations will run inside a single transaction by default. Incontrast, if a database doesn’t support DDL transactions (e.g. MySQL, Oracle)then all operations will run without a transaction.

You can prevent a migration from running in a transaction by setting theatomic attribute to False. For example:

It’s also possible to execute parts of the migration inside a transaction usingatomic() or by passing atomic=True toRunPython. SeeNon-atomic migrations for more details.

Dependencies¶

While migrations are per-app, the tables and relationships implied byyour models are too complex to be created for one app at a time. When you makea migration that requires something else to run - for example, you add aForeignKey in your books app to your authors app - the resultingmigration will contain a dependency on a migration in authors.

This means that when you run the migrations, the authors migration runsfirst and creates the table the ForeignKey references, and then the migrationthat makes the ForeignKey column runs afterwards and creates the constraint.If this didn’t happen, the migration would try to create the ForeignKeycolumn without the table it’s referencing existing and your database wouldthrow an error.

This dependency behavior affects most migration operations where yourestrict to a single app. Restricting to a single app (either inmakemigrations or migrate) is a best-efforts promise, and nota guarantee; any other apps that need to be used to get dependencies correctwill be.

Apps without migrations must not have relations (ForeignKey,ManyToManyField, etc.) to apps with migrations. Sometimes it may work, butit’s not supported.

Migration files¶

Migrations are stored as an on-disk format, referred to here as“migration files”. These files are actually normal Python files with anagreed-upon object layout, written in a declarative style.

A basic migration file looks like this:

What Django looks for when it loads a migration file (as a Python module) isa subclass of django.db.migrations.Migration called Migration. It theninspects this object for four attributes, only two of which are usedmost of the time:

• dependencies, a list of migrations this one depends on.
• operations, a list of Operation classes that define what thismigration does.

The operations are the key; they are a set of declarative instructions whichtell Django what schema changes need to be made. Django scans them andbuilds an in-memory representation of all of the schema changes to all apps,and uses this to generate the SQL which makes the schema changes.

That in-memory structure is also used to work out what the differences arebetween your models and the current state of your migrations; Django runsthrough all the changes, in order, on an in-memory set of models to comeup with the state of your models last time you ran makemigrations. Itthen uses these models to compare against the ones in your models.py filesto work out what you have changed.

You should rarely, if ever, need to edit migration files by hand, butit’s entirely possible to write them manually if you need to. Some of themore complex operations are not autodetectable and are only available viaa hand-written migration, so don’t be scared about editing them if you have to.

Custom fields¶

You can’t modify the number of positional arguments in an already migratedcustom field without raising a TypeError. The old migration will call themodified __init__ method with the old signature. So if you need a newargument, please create a keyword argument and add something likeassert'argument_name'inkwargs in the constructor.

Model managers¶

You can optionally serialize managers into migrations and have them availablein RunPython operations. This is doneby defining a use_in_migrations attribute on the manager class:

If you are using the from_queryset() function todynamically generate a manager class, you need to inherit from the generatedclass to make it importable:

Please refer to the notes about Historical models in migrations to seethe implications that come along.

Initial migrations¶

Migration.initial¶

The “initial migrations” for an app are the migrations that create the firstversion of that app’s tables. Usually an app will have one initial migration,but in some cases of complex model interdependencies it may have two or more.

Initial migrations are marked with an initial=True class attribute on themigration class. If an initial class attribute isn’t found, a migrationwill be considered “initial” if it is the first migration in the app (i.e. ifit has no dependencies on any other migration in the same app).

When the migrate--fake-initial option is used, these initialmigrations are treated specially. For an initial migration that creates one ormore tables (CreateModel operation), Django checks that all of those tablesalready exist in the database and fake-applies the migration if so. Similarly,for an initial migration that adds one or more fields (AddField operation),Django checks that all of the respective columns already exist in the databaseand fake-applies the migration if so. Without --fake-initial, initialmigrations are treated no differently from any other migration.

History consistency¶

As previously discussed, you may need to linearize migrations manually when twodevelopment branches are joined. While editing migration dependencies, you caninadvertently create an inconsistent history state where a migration has beenapplied but some of its dependencies haven’t. This is a strong indication thatthe dependencies are incorrect, so Django will refuse to run migrations or makenew migrations until it’s fixed. When using multiple databases, you can use theallow_migrate() method of database routers to control which databasesmakemigrations checks for consistent history.

Adding migrations to apps¶

New apps come preconfigured to accept migrations, and so you can add migrationsby running makemigrations once you’ve made some changes.

If your app already has models and database tables, and doesn’t have migrationsyet (for example, you created it against a previous Django version), you’llneed to convert it to use migrations by running:

This will make a new initial migration for your app. Now, run pythonmanage.pymigrate--fake-initial, and Django will detect that you have aninitial migration and that the tables it wants to create already exist, andwill mark the migration as already applied. (Without the migrate--fake-initial flag, the command would error out because the tables it wantsto create already exist.)

Note that this only works given two things:

• You have not changed your models since you made their tables. For migrationsto work, you must make the initial migration first and then make changes,as Django compares changes against migration files, not the database.
• You have not manually edited your database - Django won’t be able to detectthat your database doesn’t match your models, you’ll just get errors whenmigrations try to modify those tables.

Reversing migrations¶

Migrations can be reversed with migrate by passing the number of theprevious migration. For example, to reverse migration books.0003:

If you want to reverse all migrations applied for an app, use the namezero:

A migration is irreversible if it contains any irreversible operations.Attempting to reverse such migrations will raise IrreversibleError:

Historical models¶

When you run migrations, Django is working from historical versions of yourmodels stored in the migration files. If you write Python code using theRunPython operation, or if you haveallow_migrate methods on your database routers, you need to use thesehistorical model versions rather than importing them directly.

Warning

If you import models directly rather than using the historical models,your migrations may work initially but will fail in the future when youtry to re-run old migrations (commonly, when you set up a new installationand run through all the migrations to set up the database).

This means that historical model problems may not be immediately obvious.If you run into this kind of failure, it’s OK to edit the migration to usethe historical models rather than direct imports and commit those changes.

Because it’s impossible to serialize arbitrary Python code, these historicalmodels will not have any custom methods that you have defined. They will,however, have the same fields, relationships, managers (limited to those withuse_in_migrations=True) and Meta options (also versioned, so they maybe different from your current ones).

Warning

This means that you will NOT have custom save() methods called on objectswhen you access them in migrations, and you will NOT have any customconstructors or instance methods. Plan appropriately!

References to functions in field options such as upload_to andlimit_choices_to and model manager declarations with managers havinguse_in_migrations=True are serialized in migrations, so the functions andclasses will need to be kept around for as long as there is a migrationreferencing them. Any custom model fieldswill also need to be kept, since these are imported directly by migrations.

In addition, the concrete base classes of the model are stored as pointers, soyou must always keep base classes around for as long as there is a migrationthat contains a reference to them. On the plus side, methods and managers fromthese base classes inherit normally, so if you absolutely need access to theseyou can opt to move them into a superclass.

To remove old references, you can squash migrationsor, if there aren’t many references, copy them into the migration files.

Considerations when removing model fields¶

Similar to the “references to historical functions” considerations described inthe previous section, removing custom model fields from your project orthird-party app will cause a problem if they are referenced in old migrations.

To help with this situation, Django provides some model field attributes toassist with model field deprecation using the system checks framework.

Add the system_check_deprecated_details attribute to your model fieldsimilar to the following:

After a deprecation period of your choosing (two or three feature releases forfields in Django itself), change the system_check_deprecated_detailsattribute to system_check_removed_details and update the dictionary similarto:

You should keep the field’s methods that are required for it to operate indatabase migrations such as __init__(), deconstruct(), andget_internal_type(). Keep this stub field for as long as any migrationswhich reference the field exist. For example, after squashing migrations andremoving the old ones, you should be able to remove the field completely.

Data Migrations¶

As well as changing the database schema, you can also use migrations to changethe data in the database itself, in conjunction with the schema if you want.

Migrations that alter data are usually called “data migrations”; they’re bestwritten as separate migrations, sitting alongside your schema migrations.

Django can’t automatically generate data migrations for you, as it does withschema migrations, but it’s not very hard to write them. Migration files inDjango are made up of Operations, andthe main operation you use for data migrations isRunPython.

To start, make an empty migration file you can work from (Django will putthe file in the right place, suggest a name, and add dependencies for you):

Then, open up the file; it should look something like this:

Now, all you need to do is create a new function and haveRunPython use it.RunPython expects a callable as its argumentwhich takes two arguments - the first is an app registry that has the historical versions of all your modelsloaded into it to match where in your history the migration sits, and thesecond is a SchemaEditor, which you can use tomanually effect database schema changes (but beware, doing this can confusethe migration autodetector!)

Let’s write a migration that populates our new name field with the combinedvalues of first_name and last_name (we’ve come to our senses andrealized that not everyone has first and last names). All we need to do is usethe historical model and iterate over the rows:

Once that’s done, we can run pythonmanage.pymigrate as normal and thedata migration will run in place alongside other migrations.

You can pass a second callable toRunPython to run whatever logic youwant executed when migrating backwards. If this callable is omitted, migratingbackwards will raise an exception.

Accessing models from other apps¶

When writing a RunPython function that uses models from apps other than theone in which the migration is located, the migration’s dependenciesattribute should include the latest migration of each app that is involved,otherwise you may get an error similar to: LookupError:Noinstalledappwithlabel'myappname' when you try to retrieve the model in the RunPythonfunction using apps.get_model().

In the following example, we have a migration in app1 which needs to usemodels in app2. We aren’t concerned with the details of move_m1 otherthan the fact it will need to access models from both apps. Therefore we’veadded a dependency that specifies the last migration of app2:

More advanced migrations¶

If you’re interested in the more advanced migration operations, or wantto be able to write your own, see the migration operations reference and the “how-to” on writing migrations.

Squashing migrations¶

You are encouraged to make migrations freely and not worry about how many youhave; the migration code is optimized to deal with hundreds at a time withoutmuch slowdown. However, eventually you will want to move back from havingseveral hundred migrations to just a few, and that’s where squashing comes in.

Squashing is the act of reducing an existing set of many migrations down toone (or sometimes a few) migrations which still represent the same changes.

Django does this by taking all of your existing migrations, extracting theirOperations and putting them all in sequence, and then running an optimizerover them to try and reduce the length of the list - for example, it knowsthat CreateModel andDeleteModel cancel each other out,and it knows that AddField can berolled into CreateModel.

Once the operation sequence has been reduced as much as possible - the amountpossible depends on how closely intertwined your models are and if you haveany RunSQLor RunPython operations (which can’tbe optimized through unless they are marked as elidable) - Django will thenwrite it back out into a new set of migration files.

These files are marked to say they replace the previously-squashed migrations,so they can coexist with the old migration files, and Django will intelligentlyswitch between them depending where you are in the history. If you’re stillpart-way through the set of migrations that you squashed, it will keep usingthem until it hits the end and then switch to the squashed history, while newinstalls will use the new squashed migration and skip all the old ones.

This enables you to squash and not mess up systems currently in productionthat aren’t fully up-to-date yet. The recommended process is to squash, keepingthe old files, commit and release, wait until all systems are upgraded withthe new release (or if you’re a third-party project, ensure your users upgradereleases in order without skipping any), and then remove the old files, commitand do a second release.

The command that backs all this is squashmigrations - pass it theapp label and migration name you want to squash up to, and it’ll get to work:

Visual Studio For Mac Os

Use the squashmigrations--squashed-name option if you want to setthe name of the squashed migration rather than use an autogenerated one.

Note that model interdependencies in Django can get very complex, and squashingmay result in migrations that do not run; either mis-optimized (in which caseyou can try again with --no-optimize, though you should also report an issue),or with a CircularDependencyError, in which case you can manually resolve it.

To manually resolve a CircularDependencyError, break out one ofthe ForeignKeys in the circular dependency loop into a separatemigration, and move the dependency on the other app with it. If you’re unsure,see how makemigrations deals with the problem when asked to createbrand new migrations from your models. In a future release of Django,squashmigrations will be updated to attempt to resolve these errorsitself.

Once you’ve squashed your migration, you should then commit it alongside themigrations it replaces and distribute this change to all running instancesof your application, making sure that they run migrate to store the changein their database.

You must then transition the squashed migration to a normal migration by:

• Deleting all the migration files it replaces.
• Updating all migrations that depend on the deleted migrations to depend onthe squashed migration instead.
• Removing the replaces attribute in the Migration class of thesquashed migration (this is how Django tells that it is a squashed migration).

Note

Once you’ve squashed a migration, you should not then re-squash that squashedmigration until you have fully transitioned it to a normal migration.

Serializing values¶

Migrations are Python files containing the old definitions of your models- thus, to write them, Django must take the current state of your models andserialize them out into a file.

While Django can serialize most things, there are some things that we justcan’t serialize out into a valid Python representation - there’s no Pythonstandard for how a value can be turned back into code (repr() only worksfor basic values, and doesn’t specify import paths).

Django can serialize the following:

• int, float, bool, str, bytes, None, NoneType
• list, set, tuple, dict, range.
• datetime.date, datetime.time, and datetime.datetime instances(include those that are timezone-aware)
• decimal.Decimal instances
• enum.Enum instances
• uuid.UUID instances
• functools.partial() and functools.partialmethod instanceswhich have serializable func, args, and keywords values.
• LazyObject instances which wrap a serializable value.
• Enumeration types (e.g. TextChoices or IntegerChoices) instances.
• Any Django field
• Any function or method reference (e.g. datetime.datetime.today) (must be in module’s top-level scope)
• Unbound methods used from within the class body
• Any class reference (must be in module’s top-level scope)
• Anything with a custom deconstruct() method (see below)

Django cannot serialize:

• Nested classes
• Arbitrary class instances (e.g. MyClass(4.3,5.7))
• Lambdas

Custom serializers¶

You can serialize other types by writing a custom serializer. For example, ifDjango didn’t serialize Decimal by default, you could dothis:

The first argument of MigrationWriter.register_serializer() is a type oriterable of types that should use the serializer.

The serialize() method of your serializer must return a string of how thevalue should appear in migrations and a set of any imports that are needed inthe migration.

Adding a deconstruct() method¶

Mac Visual Studio Code

You can let Django serialize your own custom class instances by giving the classa deconstruct() method. It takes no arguments, and should return a tupleof three things (path,args,kwargs):

• path should be the Python path to the class, with the class name includedas the last part (for example, myapp.custom_things.MyClass). If yourclass is not available at the top level of a module it is not serializable.
• args should be a list of positional arguments to pass to your class’__init__ method. Everything in this list should itself be serializable.
• kwargs should be a dict of keyword arguments to pass to your class’__init__ method. Every value should itself be serializable.

Note

This return value is different from the deconstruct() methodfor custom fields which returns atuple of four items.

Django will write out the value as an instantiation of your class with thegiven arguments, similar to the way it writes out references to Django fields.

To prevent a new migration from being created each timemakemigrations is run, you should also add a __eq__() method tothe decorated class. This function will be called by Django’s migrationframework to detect changes between states.

As long as all of the arguments to your class’ constructor are themselvesserializable, you can use the @deconstructible class decorator fromdjango.utils.deconstruct to add the deconstruct() method:

The decorator adds logic to capture and preserve the arguments on theirway into your constructor, and then returns those arguments exactly whendeconstruct() is called.

Supporting multiple Django versions¶

If you are the maintainer of a third-party app with models, you may need toship migrations that support multiple Django versions. In this case, you shouldalways run makemigrationswith the lowest Django version you wishto support.

The migrations system will maintain backwards-compatibility according to thesame policy as the rest of Django, so migration files generated on Django X.Yshould run unchanged on Django X.Y+1. The migrations system does not promiseforwards-compatibility, however. New features may be added, and migration filesgenerated with newer versions of Django may not work on older versions.

See also

The Migrations Operations Reference

Covers the schema operations API, special operations, and writing yourown operations.

The Writing Migrations “how-to”

Explains how to structure and write database migrations for differentscenarios you might encounter.