More Stuff

Let's focus on several useful topics that don't fit elsewhere. Most of these topics aren't crucial for a beginner. However, you will encounter them when you read Python code, documentation, and articles. Be ready!

Some of these topics have enough depth for a separate chapter, if not an entire book! We'll stick to the basics and give you a quick introduction to each.

Function Composition

A common Python technique is composing function calls, also known as composition. Composition occurs when a function call is used as an argument to another function call, which may, in turn, be passed to another function call. In each case, the return value of the inner function call is used as the argument to the outer function. We've seen many examples like this throughout this book but haven't really remarked on it. For instance, in the Introduction to Collections chapter, we saw this code:

print(list(range(3, 17, 4)))

When Python sees code like this, it first evaluates what it can without function calls. In this case, it determines that 3, 17, and 4 are integers. With those values, the only operation that can be performed directly is to evaluate range(3, 17, 4), so that's what it does. Once it has the range object, it can use the object as the argument to the list constructor. The constructor returns the list [3, 7, 11, 15], which Python passes to print for printing.

Let's look at some other examples of composing functions. First, let's define add and subtract functions and call them:

def add(a, b):
  return a + b

def subtract(a, b):
  return a - b

sum = add(20, 45)
print(sum)                              # 65

difference = subtract(80, 10)
print(difference)                       # 70

Both add and subtract take two arguments and return the result of performing an arithmetic operation on what we assume to be numeric values.

This works fine. However, we can use function composition to use a function call as an argument to another function. Stated differently, we're saying that we can write add(20, 45) and subtract(80, 10) as arguments for another function:

print(add(20, 45))                      # 65
print(subtract(80, 10))                 # 70

Passing the function call's return value to print is a canonical Python example of function composition. It's functional but uninteresting. Things get more intriguing when you pass function call results to a function that does something more complicated:

def add(a, b):
  return a + b

def subtract(a, b):
  return a - b

def times(num1, num2):
  return num1 * num2

print(times(add(20, 45), subtract(80, 10))) # 4550
# 4550 == ((20 + 45) * (80 - 10))

Here, we pass the return values of add(20, 45) and subtract(80, 10) to the times function, and we pass the return value of times to print! It produces the same result as the following verbose code:

total = add(20, 45)
difference = subtract(80, 10)
result = times(total, difference)
print(result)

Composition works best when each of the inner functions returns an object other than None. The outermost function can return anything, including None. For instance, in print(times(add(20, 45), subtract(80, 10))), the inner functions (add, subtract, and times) all return integers, but the outermost function, print, returns None.

Most built-in functions return an object that another function can use. Thus, composition in Python is relatively commonplace. If you use composition, however, keep things simple. A deeply nested chain of function calls can be challenging to understand.

Method Chaining

Well-designed methods tend to do one thing and one thing only. For instance, the str.upper method returns a string in uppercase, and the str.split method returns the result of splitting a string into words. That means you can do something like this:

tv_show = "Monty Python's Flying Circus"
tv_show = tv_show.upper()
# "MONTY PYTHON'S FLYING CIRCUS"

tv_show = tv_show.split()
# ['MONTY', "PYTHON'S", 'FLYING', 'CIRCUS']

Method chaining is similar to function composition but applies to methods specifically rather than ordinary functions. It lets us rewrite the above code more concisely.

tv_show = "Monty Python's Flying Circus"
tv_show = tv_show.upper().split()
# ['MONTY', "PYTHON'S", 'FLYING', 'CIRCUS']

To use method chaining, you start by calling a method on an appropriate object. If that method returns another object, you can use it to call another method. We use the tv_show string in the above code to call the str.upper method. That method returns a new string in uppercase. We then use the uppercase string to call str.split to create a list of words.

You can chain as many method calls as necessary, though it can get messy if you chain more than 2 or 3 methods on a single line. Fortunately, you can format your code to make it more understandable:

letters = 'abcdefghijklmnoqrstuvwxyz'

# Note that the parentheses surrounding this
# multi-line chain are required.
consonants = (letters.replace('a', '').
                      replace('e', '').
                      replace('i', '').
                      replace('o', '').
                      replace('u', ''))
print(consonants)    # bcdfghjklmnqrstvwxyz

Chaining only works when each method in the chain except the last returns an object with at least one useful method. (It doesn't matter what the last method returns.) This is rare in Python compared to most languages; many methods simply return None. Thus, chaining in Python isn't very common. However, you will encounter it. If you use chaining, keep things simple. A long chain of different kinds of operations can be challenging to understand.

Modules

You can use code from other Python files in your programs. No, we don't mean you should copy and paste the code. Instead, this is code you can make available to your program by importing it.

These files are called modules. Once you load a module, you can use its functionality in your program. This is the chief way Python programmers reuse code and keep their programs organized.

Python provides several hundred modules with the main Python distribution. You can read about these modules here. As of summer 2023, you can find nearly a half-million additional modules at PyPI.

The modules distributed with Python are always available. You don't need to download anything; you import them and use the code. For the modules at PyPI, you must download the modules before you can import them. You can do that with the pip command, which we describe in our PY101 course.

Finally, you can write your own modules; every Python file is a module. We'll show you how to use this feature in the Core Curriculum.

With all these modules, you don't need to reinvent the wheel whenever you write code. Reinventing the wheel can be fun but wastes time on real projects. You can look for modules to use when working on real projects. However, this can be a significant challenge. On large projects, you can have people devoted entirely to choosing and managing which modules the team uses.

That said, you're learning now. With standard and non-standard modules, you can solve, partially or entirely, many of the exercises and projects we'll ask you to work on. However, please don't do that. There's little benefit to going on a module hunt when you should be learning to write your own Python code. We'll tell you where you can use modules.

The import and from Statements

The import statement is used in Python to load code from Python modules into your code. The most basic way to load a module is to write import module_name, where module_name is the module's name. The import statement(s) are conventionally coded at the very top of the program file. For example:

import math

print(math.sqrt(math.pi))   # 1.7724538509055159

This code first imports the math module and then prints the square root of the pi constant. Note that both sqrt and pi are fully qualified names: both names are prefixed with the module name and a period (.). These fully qualified names mean you don't have to worry about naming conflicts with your code or other modules. (However, the name math might cause a conflict if you aren't careful.)

If you don't want to write math. every time you use a function or constant from the module, you can import just the names you want by using the from statement:

from math import pi, sqrt

print(sqrt(pi))             # 1.7724538509055159

Thisfrom statement imports the math module's pi and sqrt identifiers into your program. You can then use those names without qualification. Be aware, though, that from circumvents the naming conflict benefits of import.

You can also use an alias to avoid having to write math. every time you use something from the math module:

import math as m

print(m.sqrt(m.pi))         # 1.7724538509055159

Module aliases generally work best when the module name is lengthy, such as configparser and multiprocessing.

With the mechanics of using modules covered, let's take a short look at a few useful built-in modules:

The math Module

Python is heavily used in the science and mathematics realms. The amount of support available for Python in these realms is breathtaking. For instance, libraries like NumPy, SciPy, and Pandas provide advanced mathematical functions and data structures. Matplotlib, Seaborn, and Plotly offer potent data visualization tools. Even if you work in a different field, you may still need mathematical and visualization tools.

Even without such sophisticated libraries, most programs must perform some arithmetic or mathematical operations. That doesn't mean you need much math as a programmer; most programs need little more than basic arithmetic like a + 1.

However, sometimes you need a bit more: you may need to calculate the square root of a number, determine the value of π (pi), round floating point numbers, or even do some trigonometric calculations. You can use well-known algorithms to make these computations, but you don't have to. The Python math module provides a collection of functions and constants you can use without a complete understanding of how they work.

Let's say you want to calculate the square root of a number. It's possible to design and implement an algorithm that calculates the root. Why bother? You can use math.sqrt without first designing, writing, and testing some code:

import math
print(math.sqrt(36))        # 6.0
print(math.sqrt(2))         # 1.4142135623730951

Perhaps you need to use the number π for something. Again, you can calculate π with some precision with one of the many algorithms. However, the math.pi constant gives you immediate access to a reasonably precise approximation of its value:

print(math.pi)              # 3.141592653589793

The datetime Module

Suppose you want to determine the day of the week on which July 16th occurred in 2022. How would you go about that? You may be able to develop an appropriate algorithm on your own, but working with dates and times is a messy process. It's often much more complex than you think.

You don't have to work that hard, however. Python's datetime module provides classes for creating and manipulating objects representing a time and date. Compared to doing it yourself, it's not hard to determine the day of the week corresponding to a date. It's not easy, however; it involves two methods with particularly opaque formatting arguments:

from datetime import datetime as dt

date = dt.strptime("July 16, 2022", "%B %d, %Y")
weekday_name = date.strftime('%A')
print(weekday_name)                   # Saturday

We first import the datetime class from the datetime module (otherwise known as datetime.datetime), and then give the class an alias of dt.

Next, we use the strptime method to create an instance of the datetime.datetime class that represents "July 16, 2022". The second argument passed to strptime describes the how strptime should treat the first argument:

• %B tells strptime to look for the spelled-out month name (July).
• %d tells strptime to look for the day of the month (16).
• %Y tells strptime to look for the 4 digit year (2022).
• Note that the remaining characters in the second argument must match the corresponding characters in the first string:
  • At least one space must follow the month name.
  • A comma and at least one space must follow the day of month.
  • There can be no other characters in the string.

Finally, we call date.strftime('%A') to obtain the spelled-out weekday name (as given by %A).

Messing with dates and times is incredibly difficult and error-prone. Even with the powerful tools provided by Python, it can be tough to figure out what you need to do.

The real benefit of datetime and some other date/time-related modules is that they abstract away most of the nasty details of working with dates and times. For a good time, search the web or use an AI to find information on some of these things:

• How are leap years determined?
• How many hours elapsed in the US between 1 a.m. and 5 p.m. on October 28, 1956? How many hours elapsed during that same period in Indiana?
• How many days were there in September 1752 in the US?
• How many days were there in February 1918 in Russia?
• When does Daylight Savings Time begin and end? How about in Australia?
• Are all time zones an hour apart?
• What time is it now in Iran? What about Nepal? Compare with your current time.
• How do Greeks spell the name of the second month of the year?
• How do Germans spell the name of the weekday the English-speaking world knows as Wednesday?
• On what day of the week does the week begin?

Your head is probably swimming by now. Rest easy, though. The datetime module takes care of all this for you.

Function Definition Order

Let's look at the function definition order. Take a look at this code:

def top():
    bottom()

def bottom():
    print('Reached the bottom')

top()

Note that top is trying to call the bottom function, but bottom isn't yet defined on line 2. Will this code work? Think about it briefly and try to figure out what will happen and why. Once you've got your answer, go ahead and run the code.

What happened? The code ran just fine! It printed Reached the bottom. How does that work?

The answer lies in how Python executes def statements. When Python encounters a def statement, it merely reads the function definition into memory. It saves it away as an object in the heap. The function's body isn't executed until it's called explicitly. In this code, this read-and-save-but-don't-execute process occurs when Python encounters both functions. When we eventually invoke top on line 7, Python knows what and where top and bottom are.

What's more, Python also knows what code those functions contain. Thus, when top tries to invoke bottom, Python only has to find and call the bottom function object. That means the code runs correctly even though bottom was defined after top was.

What will happen if we call top before defining the function?

top()

def top():
    bottom()

def bottom():
    print('Reached the bottom')

Oops. That doesn't work. It fails with a NameError: name 'top' is not defined error. It doesn't work because we're trying to invoke top on line 1 before it gets defined. It has no idea what top is, so it gives up.

These issues arise when you start writing slightly longer programs with multiple functions. Many new developers stress over the order in which they define their functions, worried that Python won't be able to find them when the program runs. In practice, you don't have to worry about it. The only rule of thumb is that you should define all your functions before you try to invoke the first one. This is why Pythonistas almost always put the main program code at the bottom of the program after defining all functions.

Nested Functions

You can create functions anywhere, even nested inside another function:

def foo():
    def bar():
        print('BAR')

    bar() # BAR

foo()
bar() # NameError: name 'bar' is not defined

Here, the bar function is nested within the foo function. Such nested functions get created and destroyed every time the outer function runs. (This usually has a negligible effect on performance.) They are also private functions since we can't access a nested function from outside the function where it is defined.

Nested functions play a valuable role in Python's power and flexibility. You will meet them again if you continue on the Python learning path.

The global and nonlocal Statements

Functions add an additional level of complexity to the concept of scope. What happens when a function assigns a variable to a new value while a variable with the same name already exists in the outer scope? Enter the global and nonlocal statements.

By default, any variable that is defined in the outer scope of a function is also available inside that function:

greeting = 'Salutations'

def well_howdy(who):
    print(f'{greeting}, {who}')

well_howdy('Angie')
print(greeting)

Python assumes that all variables that are assigned a value inside a function are local variables. Even if a variable by the same name exists in an outer scope, Python will create a new local variable if the variable's name appears on the left side of an assignment. (This is the concept of shadowing that we met earlier.)

greeting = 'Salutations'

def well_howdy(who):
    #highlight
    greeting = 'Howdy'
    #endhighlight
    print(f'{greeting}, {who}')

well_howdy('Angie')
print(greeting)

In the above code, we have a variable named greeting defined in the global (top-most) scope. It's value is 'Salutations'. However, in the well_howdy function, we have an assignment of 'Howdy' to a variable named greeting. In this case, the greeting variable inside the function is completely independent from the greeting in the top-level code; it shadows the top-level greeting. Thus, the output from the above code is:

Howdy, Angie
Salutations

Python's global and nonlocal statements let the programmer override this behavior. They tell Python to use a variable that is defined elsewhere. Python will not create a new local variable when global or nonlocal is used to override this behavior.

In the case of global, Python is told to look to the outermost scope (the global scope) for the variable to be used. It works in any function. The nonlocal statement, however, only works in nested functions: functions that are defined inside an outer function. When Python processes a nonlocal statement, it looks for the associated variable in one of the outer functions.

In the well_howdy code shown earlier, suppose we want well_howdy to use the global greeting variable. To do that, all we have to do is include a global greeting statement in the function:

greeting = 'Salutations'

def well_howdy(who):
    # highlight
    global greeting
    # endhighlight
    greeting = 'Howdy'
    print(f'{greeting}, {who}')

well_howdy('Angie')
print(greeting)

When we run this program now, we'll see this output:

Howdy, Angie
Howdy

Instead of creating a new local variable in the well_howdy function, the presence of global greeting caused Python to use the global greeting variable instead.

An interesting variation occurs when the variable named by the global statement has not yet been defined:

def set_pi():
    global pi
    pi = 3.1415

set_pi()
print(pi)

In this case, the assignment ended up creating a new global variable named pi.

Suppose instead that we had some nested functions, each with its own local variable, foo:

def outer():
    def inner1():
        def inner2():
            foo = 3
            print(f"inner2 -> {foo} with id {id(foo)}")

        foo = 2
        inner2()
        print(f"inner1 -> {foo} with id {id(foo)}")

    foo = 1
    inner1()
    print(f"outer -> {foo} with id {id(foo)}")

outer()

Here, we have an outer function that we invoke on line 15. It creates an inner1 function and then invokes it. inner1, in turn, creates an inner2 function and then executes it. Each function -- outer, inner1, and inner2 -- assigns a local variable named foo to a value, calls the nested function, and then prints the value of its version of foo along with the id of foo. As you might expect, we get the following output:

inner2 -> 3 with id 4339312328
inner1 -> 2 with id 4339312296
outer -> 1 with id 4339312264

Notice that all 3 foos have different values and ids.

As with the earlier situations, we may want some of these functions to use one of the variables defined in the surrounding scope. For instance, assume we want foo in inner2 to reference the foo variable in inner1. Maybe we can use global:

def outer():
    def inner1():
        def inner2():
            # highlight
            global foo
            # endhighlight
            foo = 3
            print(f"inner2 -> {foo} with id {id(foo)}")

        foo = 2
        inner2()
        print(f"inner1 -> {foo} with id {id(foo)}")

    foo = 1
    inner1()
    print(f"outer -> {foo} with id {id(foo)}")

outer()
# highlight
print(f"global -> {foo} with id {id(foo)}")
# endhighlight

inner2 -> 3 with id 4339312328
inner1 -> 2 with id 4339312296
outer -> 1 with id 4339312264
global -> 3 with id 4339312328     # Note: same as `inner2`

Hmm. That's not what we wanted. Instead of updating the foo inside inner1, it created a new global variable and assigned that. No matter how deeply nested we are, the global statement stipulates that the listed identifiers are to be interpreted as global variables; that is, as identifiers in the global namespace.

The answer to this problem is to use the nonlocal statement:

def outer():
    def inner1():
        def inner2():
            # highlight
            nonlocal foo
            # endhighlight
            foo = 3
            print(f"inner2 -> {foo} with id {id(foo)}")

        foo = 2
        inner2()
        print(f"inner1 -> {foo} with id {id(foo)}")

    foo = 1
    inner1()
    print(f"outer -> {foo} with id {id(foo)}")

outer()
# highlight
# print(f"global -> {foo} with id {id(foo)}")  # statement removed
# endhighlight

inner2 -> 3 with id 4339312328
inner1 -> 3 with id 4339312328       # Same as inner2
outer -> 1 with id 4339312264

Ah. This time, we managed to change foo from inner1. However, it did not change foo in outer. We can address that readily by adding a nonlocal statement in inner1:

def outer():
    def inner1():
        def inner2():
            nonlocal foo
            foo = 3
            print(f"inner2 -> {foo} with id {id(foo)}")

        # highlight
        nonlocal foo
        # endhighlight
        foo = 2
        inner2()
        print(f"inner1 -> {foo} with id {id(foo)}")

    foo = 1
    inner1()
    print(f"outer -> {foo} with id {id(foo)}")

outer()

inner2 -> 3 with id 4339312328
inner1 -> 3 with id 4339312328
outer -> 3 with id 4339312328     # All 3 are the same

This time, we got 3's across the board. Effectively, the assignment in inner2 changed the value of foo in inner2, inner1, and outer.

One final note: unlike with the global statement, the nonlocal statement requires the named variable to already exist. You can't create the variable in the function that uses nonlocal.

Summary

The information in this chapter is not on the critical path to learning Python. Still, you will definitely run into these issues, and it's better to have some exposure and familiarity now than to spend hours in confusion later on. Experiment with these exercises and watch your understanding deepen.