Variables as Pointers

This chapter examines the concepts behind variables and pointers. Specifically, we'll see how Python variables are pointers to a location in memory (an address space) that contains the object assigned to the variable. New programmers often struggle to master this concept, but the material is crucial. We'll meet these concepts again in Core.

Developers sometimes talk about references instead of pointers. At Launch School, we use these terms interchangeably. You can say that a variable points to or references an object in memory, and you can also say that the pointers stored in variables are references. Some languages make a distinction between references and pointers, but Python does not; feel free to use either term.

Variables As Pointers

The behaviors described in this chapter arise from the interaction of variables using pointers to reference their associated objects. In Python, all variables are pointers to objects. If you assign the same object to multiple variables, every one of those variables references (points to) the same object. They act like aliases for the object.

When you reassign a variable, Python changes what object the variable references. Reassignment doesn't alter the old or new object; it simply changes which object the variable references.

When a reassignment involves the creation of a new object, Python first creates the new object. It then changes the variable's stack item to point to the new object. As in the previous paragraph, this does not alter the object.

Things get a little messier when mutating objects through a variable. If a variable points to a mutable object and you do something to mutate that object, Python doesn't change the variable; it changes the object. Every variable that references that object will immediately see the object's new state.

The indexing syntax can make things a little confusing. For instance, assume we have a numbers list that looks like this:

numbers = [1, 2, 3, 4]

Now, let's reassign numbers[2]:

numbers[2] = 3333

This is not a reassignment of the numbers variable. It's a mutation of the list object referenced by numbers. Technically, it's also a reassignment of the list object element at index 2, but it is not a mutation of that element.

When using variables, it's essential to remember that some operations mutate objects while others do not. For instance, list.sort mutates a list, but sorted(list) does not. In particular, you must understand how lines 2 and 3 in the following code differ:

x = [1, 2, 3, 4, 5]
x = [1, 2, 3]
x[2] = 4

Both lines 2 and 3 are reassignments. However, line 3 mutates the list assigned to x while line 2 does not mutate anything: it simply reassigns the variable to a new value.

We should also look at augmented assignment. When the variable on the left side references an immutable value, augmented assignment acts like reassignment. Thus, all of the augmented assignments in the following code (lines 6-10) are reassignments since the values referenced on the left are all immutable:

a = 42
b = 3.1415
c = 'abc'
d = (1, 2, 3)

a *= 2
b -= 1
b /= 3
c += 'def'
d += (4, 5)

However, if the variable on the left references a mutable value, the augmented assignment is usually mutated. Thus, all the augmented assignments below mutate the value referenced by the variable:

a = [1, 2, 3]
b = {'a', 'b', 'c'}

a += [4, 5]
a *= 2
b -= {'b'}

All built-in mutable types will be mutated when used as the target of an augmented assignment. However, this behavior is not guaranteed for custom objects, which we'll meet in the Object Oriented Programming book.

It's time to jump into our first real rabbit hole: how variables and objects work in Python.

Variables and Objects

At the most basic level, variables are named locations in a computer's memory. Each variable has a unique address in memory, usually in an area called the stack, and sometimes in a different area called the heap. You don't need to understand the differences between the stack and heap at this stage, but learning that they exist will help later in Core.

We'll pretend that all variables are stored on the stack.

The space allocated for a single variable is small. On modern computers in 2023, that's typically 64 bits (or 8 bytes). Objects often far exceed that size. Thus, objects usually aren't stored on the stack. Instead, Python allocates the memory it needs for an object from the heap. Heap blocks can be pretty much any size, provided sufficient memory is available.

Once Python allocates heap space, it creates and stores the object at that location. The address of that object is then copied to the variable's stack location.

Thus, when you access a variable, Python first determines where the variable is on the stack. It then takes the object's heap address from the stack item and uses it to find the object. The variable is a pointer to a stack location, and the stack location is a pointer to the object.

Let's break that down with an example. Assume we're creating a variable named numbers and giving it an initial value of a list with 3 numbers:

numbers = [1, 2, 3]

Since numbers is a new variable, Python adds a new entry to the stack. For simplicity, we'll assume the variable's stack address is at memory location 10240.

Next, Python allocates enough memory on the heap to hold the list and its elements (actually, the list object may contain pointers to the elements, but we'll ignore that for now). Let's say that Python allocates 32 bytes at address 4344278784 for the list and its elements. It then constructs an appropriate list object with its elements at address 4344278784.

Finally, Python copies the address 4344278784 into the variable's stack address. Things now look something like this:

Suppose we now want to print the object assigned to numbers:

print(numbers)

To do this, Python looks up the name numbers in its list of variables and sees it is on the stack at address 10240. On the stack, Python can see that the object associated with numbers is at address 4344278784. Finally, it passes that address to print, which formats and prints the object.

Variables and Shared Objects

Let's assign a new variable to the same object that is referenced by numbers:

numbers2 = numbers

Since numbers2 is new, Python must create a new variable on the stack, which it does. We'll assume it's at address 10256. It then determines the memory address of the object assigned to numbers and stores that address in numbers2's stack item. That means both numbers and numbers2 now point to the same object. You can verify this by using the id function or the is operator:

print(id(numbers) == id(numbers2))      # True
print(numbers is numbers2)              # True

Note also that both the numbers and numbers2 ids reference address 4344278784 (or whatever address your Python used for the list):

print(id(numbers))            # 4344278784
print(id(numbers2))           # 4344278784

That means our variables and objects are now organized like this in memory:

The variables are in different places on the stack. However, they both reference the same object.

Pointers have a curious effect when you assign a variable that references an existing object to another variable. Instead of copying the object referenced by the variable on the right to the variable on the left, Python only copies the pointer. Thus, when we initialize numbers2 with numbers, we make both numbers and numbers2 point to the same list object: [1, 2, 3]. It's not just the same value but the same list at the same address. The two variables are independent, but since they point to the same list, that list depends on what you do with both numbers and numbers2.

Equality and Identity

Two objects, obj1 and obj2, are said to be equal when obj1 == obj2 returns True. The objects are the same object when obj1 is obj2 returns True. We can also say that obj1 and obj2 have the same identity since id(obj1) and id(obj2) return the same value when obj1 is obj2 returns True.

Consider the following code:

numbers = [1, 2, 3]
numbers2 = numbers

print(numbers)                # [1, 2, 3]
print(numbers == numbers2)    # True
print(numbers is numbers2)    # True

numbers = [1, 2, 3]
numbers2 = [1, 2, 3]

print(numbers)                # [1, 2, 3]
print(numbers == numbers2)    # True
print(numbers is numbers2)    # False

On line 5 of both examples, numbers and numbers2 are equal since numbers == numbers2 returns True. This makes sense. Both variables point to a list whose values are the same: 1, 2, and 3. As we've already seen, the lists on line 6 of Example 1 are the same objects.

However, are the lists on line 6 of Example 2 the same lists? No, they are not. They are entirely different objects even though they have the same values. As a result, line 6 prints False. Line 2 created a new object, which we assigned to numbers2. Thus, assuming that the new object is at address 4344281536, memory now looks like this:

Shallow vs. Deep Copies

When you copy objects in any language, you must understand the difference between shallow and deep copies. Most copies created by Python programs are shallow copies. That might seem strange early on, but it works out for the best. Deep copies just aren't needed all that often. The copy.copy and copy.deepcopy functions from the built-in copy module are Python's primary ways to create shallow and deep copies, respectively.

Before we explore the differences, we must first understand that the memory picture is more complex than we described earlier. Let's take this object as an example:

[[1, 2], 3, 4]

As we discussed earlier, this object gets created in the heap. However, we assumed that Python would put all 3 elements in the heap memory allocated for the list object. We now have a nested list, though. That list, [1, 2], can't be stored directly in the [[1, 2], 3, 4] list. Instead, Python allocates additional memory on the heap for the inner list ([1, 2]). Instead of storing [1, 2] in the memory allocated for [[1, 2], 3, 4], it stores a pointer to the [1, 2] object. Ultimately, the memory picture looks like this:

As you can see, we have two lists stored at different places in the heap. The outer list on the left contains 3 elements: the integers, 3 and 4, and a pointer to the second list. The inner list on the right includes integer elements, 1 and 2.

The actual memory picture is more complicated than this. The integers from both lists are also stored as separate objects. However, we're not interested in them right now, so we won't overcomplicate the diagram.

my_list = [[1, 2], 3, 4]
shallow = list(my_list)
print(shallow[0] is my_list[0])         # True

my_dict = {'abc': [1, 2, 3]}
shallow = dict(my_dict)
print(shallow['abc'] is my_dict['abc']) # True

The main takeaway is that we have two different objects. Now, let's see what happens when we try to duplicate the first list.

Shallow Copies

A shallow copy of an object is a duplicate of the original object's outermost (topmost) level. Any nested objects within the copied object aren't duplicated; they still reference the nested objects from the original object. Thus, if you mutate the nested objects in the original, those mutations will be visible in the duplicate.

For instance, consider this code:

import copy

orig = [[1, 2], 3, 4]
dup = copy.copy(orig)

print(orig is dup)           # False
print(orig == dup)           # True
orig[2] = 44
print(dup)                   # [[1, 2], 3, 4]

print(orig[0] is dup[0])     # True
orig[0][1] = 22
print(dup[0])                # [1, 22]

In this example, line 3 creates the list we talked about above. Remember, the list is actually stored as separate objects. Note that [1, 2] is referenced by a pointer stored in the original list. We then create a shallow copy of the orig list on line 4 and call the duplicate dup.

On line 6, we confirm that orig and dup reference distinct objects. However, line 7 tells us they have the same values. We then mutate orig on line 8, then, on line 9, confirm that dup hasn't changed. Remember: orig and dup are separate and distinct objects.

Line 11 shows that both orig[0] and dup[0] reference the same objects. On line 12, we use orig[0] to mutate the [1, 2] list. When we print dup[0] on line 13, we can see that it reflects the change made to orig[0].

For completeness, here's what memory looks like now. We've reduced the diagram to the bare essentials. We use [inner] to represent a pointer to the inner list.

Thus, our shallow copy only duplicated the outermost level of the original list object. The inner list remains shared between the original and duplicate lists.

Deep Copies

A deep copy of an object is an exact duplicate of the original object at the outermost (or topmost) level and every nested object, no matter how deeply nested. There's basically nothing you can do to the original object that can be seen in the duplicate or vice versa.

Let's look at what happens to our previous code example when we introduce deep copies. We've highlighted the differences:

import copy

orig = [[1, 2], 3, 4]
# highlight
dup = copy.deepcopy(orig)
# endhighlight

print(orig is dup)           # False
print(orig == dup)           # True
orig[2] = 44
print(dup)                   # [[1, 2], 3, 4]

# highlight
print(orig[0] is dup[0])     # False
orig[0][1] = 22
print(dup[0])                # [1, 2]
# endhighlight

There are no code differences besides using copy.deepcopy on line 4. Furthermore, there are no behavioral differences until we reach lines 11-13. Line 11 shows that orig[0] and dup[0] are no longer the same objects. Thus, when we mutate the [1, 2] list referenced by orig[0], we don't see any changes in dup[0].

Note that copy.deepcopy doesn't duplicate everything; in most cases, it only duplicates mutable objects. Since immutable objects don't change, there's no need to copy them - references are enough to ensure a deep copy that works.

Here's our final look at the memory picture. We now have two independent "Inner list" objects.

Should I Make Deep or Shallow Copies?

The answer to this question depends on your data structure, whether your data structure has mutable content, and whether it matters to your application.

In practice, shallow copies are frequently okay. They work best when:

• working with objects that are not collections, e.g., integers and booleans.
• working with immutable objects with no mutable components, e.g., strings.
• working with collections that have no mutable elements, e.g., tuples that don't contain mutable elements.
• needed for performance reasons. Shallow copies are always faster.
• you don't care if the mutable components of an object are shared.

You should use deep copies when shallow copies are not okay. In particular, they work best when working with collections that have mutable elements, e.g., nested lists.

Summary

The takeaway of this chapter is that Python uses pointers to decide which object a variable references. Pointers can lead to surprising and unexpected behavior when two or more variables reference the same object.