Calendar management
Embedded contextual email
Scheduling automation
Automated outreach
Telehealth
Marketplaces
Recruiting
Document management
Real estate
Communications software
Docs
SDKs
Tutorials
API status
Changelog
Support
Blog
Events
Guides
Newsroom
Partners
Build vs. buy
Customer stories
Trust & security
Why NylasNot Your Pie, MyPy!
Mypy is an instrumental tool for increasing productivity and improving products – see why (with live code examples) here.
Mypy is an instrumental tool for increasing productivity and improving products – see why (with live code examples) here.
Mypy is a tool that’s beloved at Nylas – it allows you to make your Python code type-aware. It takes the burden of knowing object types off the programmer without sacrificing the flexibility that Python’s dynamic typing system gives you.
Mypy runs separate from the execution of your code. It leverages Python’s built-in type annotations introduced in PEP 484 (extended in PEP 526) to do its static type checking.
Type annotations are independent from mypy and are just documentation without it. They can be added to your code base without using mypy and without changing how your code runs. Type annotations are supported as part of the language syntax in Python >= 3.5. For older versions of Python, type annotations have to be specified in comments, but use the same type syntax.
Mypy comes in when you actually want to validate your type annotations.
def py35_plus_example(arg1: int, arg2: MyClass) -> str: pep_526_var: str = 'syntax only in 3.6+' old_var = 'supported in all versions' # type: str return old_var
def old_py_example(arg1, arg2): # type: (int, MyClass) -> str return 'use this for python < 3.5 function annotations'
Mypy is a bit more version restrictive, so to actually run the static type checker on your code it needs to be Python 2.7 or Python 3.
When you run mypy, it only checks the expressions that have type annotations. Mypy uses gradual typing which means that expressions lacking type annotations are assumed to have the catch-all type Any. You don’t necessarily need to specify types for every function when using mypy, but the more type annotations you add, the more powerful the mypy tool becomes.
Mypy can be added to your CI pipeline, which we’ll dive into in the next section.
Mypy has grown into an integral part of our development process at Nylas and has enabled us to scale our company and code base with more confidence.
Adding mypy type annotations has made our code easier to understand. The best way to illustrate this is with an example!
Imagine you want to write a program to track what time your lunch arrives each day. For this program you’re going to need a db to store the times, a function to parse the time from the lunch order, a function to add a time and a function to get the average arrival time.
Variables and objects can have cleaner and more succinct names
def parse_string_time_from_lunch_order_obj(lunch_order_obj):
datetime_arrival_time = lunch_order_obj.arrival_time
if datetime_arrival_time is None:
return None
return convert_from_datetime_to_int(datetime_arrival_time)
The example above is a mouthful! The functions and variables have long names because they are trying to convey both what they are doing and what type of objects they are dealing with. Instead, we can use types to convey this latter.
def parse_time_from_order(order: LunchOrder) -> Optional[int]:
time = order.arrival_time
if time is None:
return None
return convert_to_int(time)
Types make reading functions easier because you just have to pay attention to inputs and outputs
def get_order_arrival_time(order):
time = get_arrival_time_from_cache(order)
if time is not None:
return time
if order.day == 'monday':
return get_arrival_time_from_first_db(order)
if order.day in ['tuesday', 'wednesday', 'thursday']:
return ask_reception(order) return yolo(order)
This function doesn’t look too bad — it’s just trying to look up the order arrival time a few different ways. Now, if I were to use this function in my code, the first thing I would want to know is the argument and return types.
In order to determine that, I would have to dive into the nooks and crannies of all four functions get_arrival_time_from_cache, get_arrival_time_from_first_db, ask_reception, and yolo. This process is immensely simplified with types.
def get_order_arrival_time(order: LunchOrder) -> int: # I don't have to care much about the internals of the function because # I know what to pass in and what will be returned
Reduces the need for None and isinstance checks
def calculate_average_arrival_time(orders):
assert orders is not None and isinstance(orders, list)
if len(orders) == 0:
return None
total_arrival_time = 0
for order in orders:
arrival_time = get_order_arrival_time(order)
assert arrival_time is not None and isinstance(arrival_time, int)
total_arrival_time += arrival_time
return total_arrival_time / len(orders)
Without types, we have to ensure that the argument orders is both present and the type that we expect it to be. There is nothing stopping some other function from calling calculate_average_arrival_time with orders as None, a string, or any other unexpected type. Without mypy, we need to have code to detect and handle unexpected arguments.
Below is the same code with mypy. Using mypy tightens up your code by making you more confident about the inputs and outputs of functions. This allows you to remove checks related to typing.
def calculate_average_arrival_time(orders: List[LunchOrder]) -> Optional[float]:
# We can remove the first assertion
# because mypy ensures the argument will be a List of LunchOrders
if len(orders) = 0:
return None
total_arrival_time = 0
for order in orders:
# We know that get_order_arrival_time always returns an int
arrival_time = get_order_arrival_time(order)
total_arrival_time += arrival_time
return total_arrival_time / len(orders)
Mypy helps keep your code clean and easy to use by making it obvious when code could be better. When annotations get annoying or painful, it’s a good sign that there is room for improvement. For example, a type annotation like List[Dict[str, Optional[Union[List[str]]]]] is a good indicator that you should probably have some decoupling or some type of object to encapsulate the complexity of a data structure. Is your code base guilty of this?
It might be counter intuitive that doing more (i.e. adding type annotations for mypy) can actually lead to more productivity, but in this case it’s true. Mypy helps you catch small errors faster. While one could argue that these errors would likely have been caught by running your tests, the process of running your whole test suite, waiting for it to blow up, having to debug and implementing the resulting fix takes time. Mypy runs much faster than running your test suite and increases developer velocity by shortening the loop between error made and error discovered and fixed.
We’ve already talked about how mypy can help catch bugs from refactors, but it also can help when finding usages of an object. Type annotations for mypy surface more usages of a specific type when compared to searching without type annotations. As an email API company, the Message class is used pretty widely in our code base. Doing a search for Message without type annotations only yields 450 results. When type annotations are included, we can find more places that interact with that class and there are 645 results; a 30% increase!
Mypy also integrates with some IDEs. For example, PyCharm understands mypy type annotations and they dramatically improve itsfunctionality for finding uses, real-time linting, and autocomplete suggestions.
When using a dynamically-typed language on large code bases, refactors can quickly become unwieldy and create bugs that are hard to unit test for. There are also bugs that eventually creep up from human error trying to mentally track types of objects. Mypy can catch all these bugs for you, so that these silly bugs don’t make it to production. The better your type annotation code coverage is, the more accurate mypy will be in catching these.
If you want to use mypy to check the type annotations in your project, you can use the prepackaged mypy tool. Any time you want to check annotations of a file or folder, you can manually run mypy /path/to/file/or/folder from your command line. Running this would output the mypy errors directly in your terminal. Below, we are running mypy on the file event_queue.py and second line shows the mypy error output.
$ mypy event_queue.py event_queue.py:40: error: No return value expected
This works for small scale projects and teams, but it’s a pretty manual process and makes it hard to ensure that all team members are running this command for every code and using the agreed upon flags.
At Nylas, we have built mypy into two steps of our development process. This takes the burden of remembering to type check off of individual engineers. In order to add mypy more seamlessly into the Nylas development process, we have developed some mypy helper tools which are open source and can be found in the nylas mypy-tools repo (https://github.com/nylas/mypy-tools).
We use the Arcanist command line tool to lint, test, and submit our code to Phabricator for code review. Arcanist is a client for Phabricator that allows you to interact with Phabricator from the command line. Arcanist allows you to subclass the ArcanistLinter and create your own custom linter. In order to add mypy into our linting process, we created a custom mypy linter. This linter runs alongside the other configured linters when developers call arc lint to lint their code or arc diff to submit their code for review.
At a high level, the mypy linter does two things: ensures new or modified functions have type annotations and verifies that these type annotations are correct.
Checking that any new or modified functions have type annotations
This is an important first step for a mypy linter because using mypy to ensure that type annotations are correct depends on the fact that type annotations exist in the first place. The linter goes through all new or modified lines and checks that their functions have type annotations. If any of these functions are missing type annotations, the linter emits errors in the console with the problematic line numbers. If there are missing annotation errors, the linter exits and blocks both the remaining linters from running, and the engineer from submitting their code for review. This means that any new or altered code must have type annotations before it can be properly linted and submitted for review.
Only requiring engineers to add annotations for code that they touched allows us to incrementally enforce adding type annotations to our codebase in a manageable way. We made this decision to ensure that our type annotation coverage would not regress, seeing that the power of mypy gets stronger as you cover more of your code base.
Running mypy to check that those type annotations are correct
In order to check that the type annotations are correct, the linter gathers all of the file paths touched and passes each through the lintPaths function. lintPaths first checks if you have the mypy server running. The mypy server speeds up type checking by maintaining a module dependency graph that it can quickly reference when it detects changes. There is a more modern version of the mypy server called dmypy — this mypy server was created before dmypy existed.
If you have the server running, lintPaths makes a request to the server and gets the mypy output as the response. If you don’t have the server running, then lintPaths will run the prepackaged mypy tool on the file passed in and parse the output. The linter has some added functionality — it only lints each path once, ensures each error is reported once, and pretty prints errors with additional context.
We have a linter rule set so that no new code with mypy errors can be submitted. The benefits of using this linter are that it automatically runs mypy on every file touched and prevents incorrectly typed files from being submitted. This helps engineers fix mypy errors that they might have introduced before submitting their code to be reviewed.
You can check out the code for our custom mypy linter in MypyLinter.php.
Once code is submitted for review, we use Jenkins to automatically run jobs to test that code. One of these jobs runs mypy on the whole project, including the new code patch. It’s different from when mypy is run locally because it will now run on all files in the project rather than just added or modified functions. This step is a catch all for mypy errors that could have been introduced.
To run mypy on the project, we use MypyTask from nylas mypy tools. MypyTask encapsulates running mypy and passing the configs in a single invocation. It executes the mypy command on the project root in a subprocess.
Adding these two steps to our development process has allowed us to add mypy to our existing codebase and increase code coverage in a manageable way.
There is a steep learning curve for using mypy and adding type annotations. When new engineers start on the team, it can be overwhelming to have to add type annotations to every function touched. Just when you think you’re done and ready to submit your code for review, the mypy linter strikes back and prints a long list of missing annotations or type errors that you have to go back and fix.
It also takes time to get familiar with the syntax and the types available. This can be discouraging and make engineers dislike mypy in the beginning. But over time, adding type annotations becomes part of your natural work flow — adding them as you go rather than at the end makes it feel like much less of a burden. You also get faster as you get more accustomed to writing them. This comes partly from personal experience and partly because as more types are added to the codebase, there are more examples to reference. You also get quicker at reading mypy error messages and more accustomed with how to handle corner cases. You eventually realize that adding types actually helps you understand the functions you are working with more clearly. This is the appreciation curve. What used to seem tedious and confusing becomes quick and informative. Pretty quickly, mypy cynics turn into mypy advocates.
In order to lessen the learning curve, make sure new team members have thorough guides for mypy syntax, commonly used and custom types in the code base.
TypedDict to Model Structural TypesTypedDict lets you define precise types for dictionaries with fixed schemas. This is helpful for tightening a dictionary type beyond Dict[str, Any]. TypedDict lets you specify specify key strings and value types. This means if you have a very specific schema for a dictionary, you can use TypedDict to ensure that variables adhere to this schema.
First, install mypy_extensions, which is officially supported by the mypy authors, with pip in order to use TypedDict. Then you can import it from mypy_extensions and use it like:
from mypy_extensions import TypedDict
Person = TypedDict(
'Person', {
'first_name': str,
'last_name': str,
'age': int
})
new_person = {
'first_name': 'John',
'last_name': 'Doe',
'age': 30
}
# type: Person
If you tried to pass a dictionary with different keys or value types to a function expecting a Person, mypy would raise an error.
Adding type annotations for mypy can sometimes introduce circular imports that wouldn’t exist otherwise. The mypy authors were aware of this, so they added a TYPE_CHECKING flag and the ability to use string names of types as forward references. The TYPE_CHECKING flag is True when mypy is running on your code, but False during normal execution, so it’s safe to put imports for type checking that would otherwise cause circular imports here.
Here’s an example of when you would need to use these two features to prevent a circular import:
movie_star.py
from typing import TYPE_CHECKING
if TYPE_CHECKING:
# True when mypy executes, but not normal execution
from movie import Movie
class MovieStar:
def get_movies(self) -> List['Movie']:
return self.movies
movie.py
from typing import TYPE_CHECKING
if TYPE_CHECKING:
# True when mypy executes, but not normal execution
from movie_star import MovieStar
class Movie:
def get_stars(self) -> List['MovieStars']:
return self.movie_stars
You can learn more about Nylas and how we employ Mypy on our github!
Please enter your email address and receive the latest updates.
Annie grew up in the Bay Area and studied CS and Psychology at Yale. She enjoys knitting and jogging slowly through Golden Gate Park.
