A matrix transpose is just swapping rows and columns, right? You take a matrix A, turn its columns into rows, then the new matrix is called A transpose. But there is way more to it than just moving things around.

In fact, we can define the transpose of a linear map without ever referring to the matrix at all. A deeper understanding of how transpose came about will help us visualize it, and along the way, we will come across deeper topics like covectors, duality, and so on.

We will first go through a very general big picture of what transpose really is, then try to visualize something called covectors, then transpose, giving some examples; and finally, a very subtle but also important point that links to special relativity.

The starting point is the concept of a covector, or dual, or 1-form, because in different contexts, different names are used, but I’m going with “covectors” here.

Covectors are basically machines that will eat a vector and output a number. I like to call this machine a measuring device – it “measures the vector.” These are special devices in that they are “linear.

” That means if you have two vectors v_1 and v_2, and they are measured by this machine to be a_1 and a_2 respectively, then this machine will measure v_1 + v_2 to be a_1 + a_2; and if you instead scale the vector v_1 by a number lambda, then it will be measured to be lambda a_1.

Now let’s say we have a linear transformation, alpha, transforming a vector v to another vector w. To emphasize that v and w can be very different, let’s say v is a 2-dimensional vector, and w could be a 3-dimensional one.

And for some reason, you want to measure w. A bit of a silly analogy would be: you want to measure your own weight, and the vector is you here.

But you don’t have a bathroom scale and you have to use your neighbor's. So one way to do it would be to go to your neighbor’s house, then take the measurement there directly.

But you could also just call the neighbor to lend you the scale, so that you can take the measurements at home, and get the same answer. The scale being moved from the neighbor’s house to yours is the transpose of alpha.

So alpha is you going to the neighbor’s house, and alpha transpose is moving the measuring device from the neighbor’s to yours.

But how is this related to the matrix transpose that you know? A matrix is, at its core, only a summary of a linear map. For a 2-dimensional linear map for instance, the first column tells you where the vector (1,0) goes; and the second column tells you where the vector (0,1) goes.

If we then want to know where an arbitrary vector (x,y) goes, just multiply the first column by x, and add to y times the second column!

Just knowing these four numbers is enough to determine the entire linear map, so the matrix is essentially the summary of the linear map.

So you have the linear transformations alpha and its transpose, and also the two matrices that are summaries of the transformations. Because the matrix on the right summarizes the transpose of alpha, it makes sense to call it Aᵀ.

It just turns out that the summary matrices of alpha and alpha transpose can be obtained by swapping rows and columns of each other.

The most important point here, though, is the big picture: a transpose is just transforming some measurement device in other spaces back to that in the original.

In what remains, we will, first of all, try to visualize those covectors, and then the transpose.

[CHAPTER 2: VISUALIZING COVECTORS] Covectors are linear, so just like our linear transformations, in the 2-dimensional case, it suffices to know what (1,0) and (0,1) are measured.

If the measurements are a and b respectively, for our purpose, let’s say this is an (a,b) covector.

Now if you take any vector (x,y), which is just x times (1,0) plus y times (0,1), just by linearity, it will be measured as x * a + y * b.

So in the 2-dimensional case, an (a,b) covector measures the vector (x,y) to be xa + yb.

This is just a function of two variables.

A natural way to visualize this would be to plot a 3-dimensional graph, with the z-coordinate being the function value at (x,y). In this case, it is just a giant plane passing through the origin.

However, because we might want to consider a 3-dimensional covector later on, we use another slightly roundabout way of visualizing such a function.

Instead, we consider its level sets. That means you plot all the points (x,y), such that it gets measured as a number c. In this specific case, we plot the line graph ax + by = c, where a is 2, and b is 3, so this represents a (2,3)-covector.

By varying c, we get a glimpse of how this function behaves.

In this case, it just moves up and down. We label this line to be 1, because that’s all the points that will be measured to be 1. And we only draw a line graph for, say, integer values of c, so we have parallel lines at regular intervals.

Of course, this is just one covector. Different covectors will have different sets of parallel lines with different directions and densities.

But now the question is, given a particular set of parallel lines, can we quickly spot which covector this is? In this case, it is the (-1, -0.5)-covector, so let’s compare with the (-1, -0.5) vector and see what happens.

The corresponding vector is perpendicular to all the lines. So, okay, given these parallel lines, all we know is that the corresponding vector lies on this perpendicular line, but why is this exact vector, then?

If we scale this vector, we notice that the longer the vector, the denser the parallel lines are.

So we expect the length to be proportional to the density of the parallel lines. More quantitatively, if we have the gap size between the lines, then the density of the lines should be 1/gap size - how many gaps could you fit in 1 unit?

And indeed, the length of the vector is exactly 1/gap size.

Okay, so you know the length, but these two vectors have the same length. How do you determine which one?

It’s actually impossible to tell unless you label back those lines. This line, for example, was labeled 2, because they are all the points where the covector would measure them to be 2.

With these labels, we can tell these two vectors apart. The corresponding vector points towards positive labels. So here is a summary of how you know which covector you have just by looking at these lines.

First off, you know the corresponding vector is perpendicular to the lines, so the vector must be on this white line. You take a look at the gap size between the lines, then you know it’s one of these two vectors whose length is 1/gap size.

Then you take a look at the hidden labels of the lines, and you choose the one pointing towards the positive labels. Finally, you read off the coordinates of the corresponding vector.

So we have visualized 2-dimensional covectors using parallel lines. 3-dimensional covectors are similar, except we have parallel planes instead. The corresponding vector would still be perpendicular to the planes, the length
of the vector would still be the density, 
or 1/gap size of those parallel planes, and the
vector still points towards the positive labels.

Now that we have visualized covectors,
we are ready for the transpose.

As we have seen in the big picture of transpose,
alpha transpose transforms a measuring device, 
a covector, in the other space back to the
original space. So we transform a covector, to another covector with the transpose.

The trouble is that one covector already 
occupies quite a lot of space. Two covectors
appearing together is already the maximum.

So we can only visualize the transformation of two
covectors at once. Which two covectors should we 
choose? Just like normal linear transformations,
the two representatives are the (1,0)-covector, as well as the (0,1)-covector.

Like a normal vector transformation, 
we observe where the (1,0)-covector goes.
Whichever covector it becomes would be our first column of the transpose matrix.

Similarly, we
observe the transformation of the (0,1)-covector. 
So for a covector transformation, we can similarly
write down a matrix representation. The first column tells you where the (1,0)-covector
goes, and the second column tells you where 
the (0,1)-covector goes.

In what follows, we
will use the right shear as an example. What should the transpose of a right shear look like?

We will need to go back to this big picture. In this specific case, alpha is a right shear,
and we now just have a 2D to 2D transformation.

Now, this (1,0)-covector measures the sheared
vector. And the job of the transpose is to find out which covector could we use to directly
measure the original vector, such that we get 
the same measurement at the end.

Now go back
to the visuals: a right shear would transform the basis vectors as such.

Next, we see how the
(1,0)-covector measures our sheared vectors. In this case, the measurements are both 1.

But rather than thinking about their exact measurements, just notice the tips all lie on this line. Remember that each line represents the tips of those vectors whose measurements are the same fixed constant c.

So in this visualization, which line the tip of the vectors are on tells you the measurement of the vectors.

This is why the transpose does this: first off, the basis vectors have gone back to their original place. And the tips of both vectors still lie on the bold line, which means that you get the same measurements as before.

This is exactly what a transpose should do! Anyway, let’s see what the transpose does to the (0,1)-covector instead. Again, we do the right shear to the basis vectors, then use the (0,1)-covector to measure them.

This time, their measurements are different. I use the bold pink line to indicate the measurement of the pink vector, and the bold green line to indicate the measurement of the green vector.

For the transpose, we want to bring back the basis vectors and retain the measurements. This time, we don’t need to do anything to the covector, because the tip of the pink vector still lies on the pink line, and the tip of the green vector still lies on the green line.

Their measurements are not changed when the vectors go back to their original place.

So altogether, this is the situation after you have done the right shear to your basis vectors.

The transpose wants the basis vectors to go back to the original place while retaining the measurements, and so it essentially does a left shear to the whole picture. Wait… originally, we have a right shear, and somehow the transpose is a left shear?

While it *looks* like a left shear, the gridlines are actually made up of two separate, different covectors.

The original right shear has the matrix (1,1; 0,1), because the (1,0) vector stays put, and it is still (1,0) after the shear, but the (0,1) vector has sheared to the (1,1) vector.

Compare this with the transpose. This is the
(1,0)-covector. The so-called “left shear” transforms such a covector to this one.

Which
covector is this? We know the corresponding
vector lies on this perpendicular line, then
we need to figure out the gap size. With the help of this vector telling us how long 1 unit is,
the gap size can be calculated to be 1/√2.

So because the length of the corresponding
vector is 1/gap size, the length is now √2.

This leaves us with two choices: either a (1,1)
or a (-1,-1) covector. Because originally, this bold line is measured positively by this
(1,0)-covector, after the transformation,
this bold line indicates the positive
direction, and so it must be the (1,1)-covector.

On the other hand, the (0,1)-covector
is unaffected by the left shear,
so it stays a (0,1)-covector. So this so-called
“left shear” should be represented by this matrix, because the first column tells us that
the (1,0)-covector has transformed to
this (1,1)-covector, and the (0,1)-covector has
just stayed put, still being the (0,1)-covector.

Now compare this matrix of this so-called
“left shear” with the original right shear.
You can see that the matrices are indeed
just swapping rows and columns of each other!

We will have more examples later
on, but this hopefully convinces you that
transpose *looks like* inverse, but you have
to treat those gridlines as covectors instead.

The first of the two other examples I would give
would be a diagonal matrix, such as the one with
entries 3, 2 on the diagonal. The first column
just tells us to enlarge the pink vector by a factor of 3, and the second column tells us to
enlarge the green vector by a factor of 2.

As before, we bring on the covectors and do the inverse transformation to get the basis vectors to where it started.

But when viewed as covectors, this is actually a (3,0)-covector, because the gap size has now reduced to 1/3 the original size.

The length of the corresponding vector is actually 3.

Exactly the same story for the other covector, except the gap size is just halved, so the length of the corresponding vector is 2. So we end up with a (0,2)-covector. This is why this transformation is still represented by the same matrix, because we view those gridlines as covectors.

And indeed, if you do the swapping rows and columns thing, you still get the same matrix!

Another example would be a rotation. Again, we start off with these basis vectors, and let’s say we rotate anticlockwise by 60 degrees. This is our linear transformation. As usual, we bring on the covector grid and do the inverse transformation to get the basis vectors to where it started.

This time, we rotate clockwise by 60 degrees. What is this covector then? Well, it is from rotating this original covector clockwise by 60 degrees, and this is the original corresponding vector.

During the rotation, the corresponding vector rotates the same way, clockwise 60 degrees.

This is because the gap size hasn’t changed just by rotation, so we don’t need to change the length of the corresponding vector. The same story for the other covector. So in this case, the transpose doesn’t just “look like” inverse, it *is* the inverse when acting on the corresponding vector.

That means, if I write down the matrix corresponding to anticlockwise 60 degree rotation, and the corresponding transpose, in this case, are inverses of each other.

In other words, for rotation, A^T = A^(-1). In general, matrices satisfying this are called orthogonal, so rotation matrices are orthogonal.

By now, you might be thinking, this visualization depends on having an inverse. And yes, indeed, there is a bit of limitation when it comes to singular matrices, and in general, in technical terms, the rank is smaller than the dimension of your original space, but those are all the cases where this visualization fails.

And even in those cases, you can still sort of displace your linear maps a tiny little bit to do the visualization, so this really is a powerful visualization for transposes.

[CHAPTER 4: SUBTLETIES] I want to emphasize a very subtle and technical but important point. The orthogonality condition Aᵀ = A⁻¹ should not have made any sense to you.

On the left hand side, the transpose is supposed to transform covectors to covectors, but on the right, A⁻¹ transforms vectors to vectors.

They aren't even the same type of object, how can they be equal?

Well, the trick is to pair up vectors and covectors, treating them sort of the same, which is what we have done in this video.

For this (a,b)-covector, where we measure (1,0) as a, (0,1) as b, we have associated with it the vector (a,b), or in the visualization of covectors as a bunch of parallel lines, we have associated it with the vector perpendicular to the lines, and length being the density of the lines.

Of course, we can do the other way around as well - given the vector, we can draw those parallel lines, and given the vector (a,b), we can also easily construct that covector.

That's how we have achieved the one-to-one correspondence between vectors and covectors.

A technical remark is that this has to be a linear correspondence, which our pairing is, but we don't have to go into the details of what linear means in this context.

Let's list a bunch of vectors and covectors that we have. If a transformation transforms a covector epsilon, to another covector eta; and epsilon is paired up with the vector v, eta with the vector w, then it looks *as if* v is being transformed to w.

This covector-to-covector transformation does not even know the existence of v, but because of this pairing, we can treat this as a vector-to-vector transformation.

In a similar way, if a vector-to-vector transformation transforms v to w, we can think of it as transforming from epsilon to eta.

It is in this sense that we say something like A^T = A^(-1).

There are two reasons why I want to emphasize the process of pairing up. First, while this way of identifying a vector with a covector is very standard and very natural, this is not the only way of doing it.

You could have also paired up with the vector (-a, b) instead, or (a, -b). These will still yield a 1-1 correspondence. There are many other options, but I’m choosing these two simply because either of these two will give you the mathematics of special relativity in 1+1 dimensions, and any matrix that fulfills this orthogonality condition, under this pairing, are the Lorentz transformations.

Don’t worry if any of these doesn’t make sense - I’ll probably make videos on special and general relativity later on.

If you know something about relativity and are wondering why this has anything to do with it, note that the act of going from vectors to covectors is the job of the metric tensor, or metric for short; and going from covectors to vectors is the job of the inverse metric.

Anyway, the point is, it is very important to specify which 1-1 correspondence, or the metric you are using, because different metrics yield different mathematics.

The deeper meaning of matrix transpose

Hi, I am Trevor. This channel discusses maths (or physics) concepts that I am passionate about, and tackles them in a novel perspective. For example, most people introduced Green's functions with lots of DE concepts, but my video on the topic did so from electrostatics, providing a vivid image of point charges. The new perspectives are not meant to be "easier" to understand, and I have never intended the videos to be pedagogical; the videos are just some interesting concepts and arguments you have not heard before on YouTube (maybe anywhere). 

For those who are wondering, I don't use coding to produce my videos, and while stylistically similar to 3Blue1Brown, I don't use Manim (Grant's mathematical animation Python library) to produce my videos. I actually use PowerPoint, GeoGebra and (sometimes) Mathematica to produce my videos.


The deeper meaning of matrix transpose

The deeper meaning of matrix transpose

Scripts