Comprehending comprehensions to write cleaner, faster Python

https://ift.tt/yBdIGlr Photo by Jonathan Cooper on  Unsplash Write better comprehensions using functions and walruses TLDR House tra...

https://ift.tt/yBdIGlr

Photo by Jonathan Cooper on Unsplash

Write better comprehensions using functions and walruses

TLDR

1. House transformation logic in functions.
2. Call the function in the list comprehension.
3. Use the walrus operator to assign variables inside the list comprehension namespace

The problem:

List and dictionary comprehensions are cornerstones of intermediate to advanced Python. Unfortunately, they are also easily misunderstood as a one-to-one replacement for for loops when you want to produce an iterable.

The comprehension syntax makes it challenging to write clear code when simply transferring the contents of the for loop into a comprehension.

Getting started:

Let’s start with a problem with a moderate amount of complexity.

For a given list of islands produced by a random name generator:

1. Remove the white space
2. Replace the word “The”
3. Make the first letter of the new word lower case

A beginner using a for loop might approach the problem like this:

island_names = ['The Neglected Holm', 'The Barnacle Key', 'The Dancing Enclave',
                'The Peaceful Island', 'Allerport Reef', 'Cresstead Archipelago',
                'Petromeny Islet', 'Esterisle Peninsula', 'Traygami Cay',
                'Savaside Peninsula']

new_island_names_loop = []

for island in island_names:
    new_island = island.replace(' ', '')
    new_island = new_island.replace('The', '')
    # turns first letter into lowercase
    new_island = new_island[0].lower() + new_island[1:]
    new_island_names_loop.append(new_island)

This code works, and is logically laid out, but isn’t particularly performant or Pythonic.

Incomprehensible comprehension:

Strictly replacing it with a list comprehension will produce this piece of garbled nonsense:

island_names = ['The Neglected Holm', 'The Barnacle Key', 'The Dancing Enclave',
                'The Peaceful Island', 'Allerport Reef', 'Cresstead Archipelago',
                'Petromeny Islet', 'Esterisle Peninsula', 'Traygami Cay',
                'Savaside Peninsula']

# This is incomprehensible and no one sane would approach the problem this way

new_island_incomprehensible_comprehenson = [
    island.replace(' ', '').replace('The', '')[0].lower() +
    island.replace(' ', '').replace('The', '')[1:]
    for island in island_names]

To understand how to fix this, we have to understand the origins of comprehensions. Python borrowed list and dictionary comprehensions from Haskell, which is a very opinionated functional language. To effectively use them, you need to think functionally. Pass each item of the iterable to a function to perform the transformation.

Functional comprehension

In this code, I encapsulated my transformation logic inside a function and call the function in the comprehension, separating the logic from the implementation:

island_names = ['The Neglected Holm', 'The Barnacle Key', 'The Dancing Enclave',
                'The Peaceful Island', 'Allerport Reef', 'Cresstead Archipelago',
                'Petromeny Islet', 'Esterisle Peninsula', 'Traygami Cay',
                'Savaside Peninsula']

# Understanding that list comprehensions come from functional languages
# helped me realize you're supposed to define a function for complex # transformations
# This also helps improve code quality by putting the transformation logic inside
# a function

def prep_island_name(island):
    new_island = island.replace(' ', '')
    new_island = new_island.replace('The', '')
    return new_island[0].lower() + new_island[1:]

new_island_comprehension = [prep_island_name(island) for island in island_names]

Here, we have the clarity of our original loop, but get to take advantage of the increased performance of the comprehension.

This is also a good practice to use for loops as well. Separating the logic from implementation makes the code more modular.

What if we need to use the same function inside the comprehension?

What if our problem requires preserving multiple steps of the transformation, using the same function multiple times?

Let’s change our prompt a little.

Now we want to take our list of islands and:

1. Remove the white space
2. Replace the word “The”
3. Attach the cleaned-up string as the key in a dictionary
4. Attach the same string with the first character turned to lowercase as the value.

A beginner may approach the problem similarly as before, using a for loop:

island_names = ['The Neglected Holm', 'The Barnacle Key', 'The Dancing Enclave', 'The Peaceful Island',
                'Allerport Reef', 'Cresstead Archipelago', 'Petromeny Islet', 'Esterisle Peninsula', 'Traygami Cay',
                'Savaside Peninsula']

new_island_dict = {}
for island in island_names:
    new_island = island.replace(' ', '')
    new_island = new_island.replace('The', '')
    # turns first letter into lowercase
    new_island_dict[new_island] = new_island[0].lower() + new_island[1:]

Similar to our earlier loop, this is easy to understand, logically laid out, and cumbersome.

Using a functional comprehension

We’ll make a couple of changes to our function from before. We’ll break up our logic into two functions, one for replacing the unwanted characters, and one for converting the first character to lowercase:

island_names = ['The Neglected Holm', 'The Barnacle Key', 'The Dancing Enclave',
                'The Peaceful Island', 'Allerport Reef', 'Cresstead Archipelago',
                'Petromeny Islet', 'Esterisle Peninsula', 'Traygami Cay',
                'Savaside Peninsula']

def prep_island_name(island):
    new_island = island.replace(' ', '')
    new_island = new_island.replace('The', '')
    return new_island

def turn_first_letter_lowercase(string):
    return string[0].lower() + string[1:]

# One challenge of using a dictionary comprehension is that we can end up making
# the same function call multiple times.
new_island_comprehension = {prep_island_name(island): 
turn_first_letter_lowercase(prep_island_name(island))
                            for island in island_names}

The logic makes sense, but we have to call prep_island_name() twice, which is messy and inefficient.

Walrus to the rescue

The walrus operator := (also called the assignment operator), introduced in Python 3.8, allows you to define variable names in namespaces you can't traditionally use = .

Now instead of calling the prep_island_name() function twice, we can use the assignment operator to define a variable inside the comprehension and pass that variable to turn_first_letter_lowercase(). Pay special attention to the parentheses that surround the key definition in our comprehension:

island_names = ['The Neglected Holm', 'The Barnacle Key', 'The Dancing Enclave',
                'The Peaceful Island', 'Allerport Reef', 'Cresstead Archipelago',
                'Petromeny Islet', 'Esterisle Peninsula', 'Traygami Cay',
                'Savaside Peninsula']

def prep_island_name(island):
    new_island = island.replace(' ', '')
    new_island = new_island.replace('The', '')
    return new_island

def turn_first_letter_lowercase(string):
    return string[0].lower() + string[1:]

# by using a walrus operator we can call the function once and then assign it a
# variable name in the comprehension name space, avoiding redundant function
# calls
new_island_comprehension_walrus = {
    (new_island := prep_island_name(island)): turn_first_letter_lowercase(new_island)
    for island in island_names}

Conclusion

It took me an embarrassingly long amount of time to figure out how to use list comprehensions effectively because I was stuck thinking of them as for loops. Thinking functionally helped me break my for loop habit and write cleaner faster code.

Learning the walrus operator strips away the final advantage of for loops, easy variable assignment.

About

Charles Mendelson is a Seattle-based Data Engineer who is also an instructional assistant at the University of Washington’s School of Professional and Continuing Education, where he teaches in their Python certificate program.

He is about to graduate with a master’s from the Harvard Extension School where he has been studying psychology. If you want to get in touch with him, the best way is on LinkedIn, where he was named one of the top 25 Data Engineering influencers of 2022 by Databand.ai.

Originally published at https://charlesmendelson.com on March 20, 2023.

---

Comprehending comprehensions to write cleaner, faster Python was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

from Towards Data Science - Medium
https://towardsdatascience.com/comprehending-comprehensions-to-write-cleaner-faster-python-d18908b42c84?source=rss----7f60cf5620c9---4
via RiYo Analytics