I keep learning new bits of linear algebra all the time, but I’m always hurting for a useful reference. I probably should get a good book (which?), but in the meantime I’m collecting several nice online sources that ML researchers seem to often recommend: The Matrix Cookbook, plus a few more tutorial/introductory pieces, aimed at an intermediate-ish level.

Main reference:

- The Matrix Cookbook – 71 pages of identities and such. This seems to be really popular.

Tutorials/introductions:

- Zico Kolter’s linear algebra review and reference [link#2]- it seems to introduce all the essentials and has very nice visual intutions for some things. (May 2015 update:) There’s now a nice video course too to go with it. (26 pages)
- Minka’s Old and New Matrix Algebra Useful for Statistics – has a great part on how to do derivatives. (19 pages)
- MacKay’s The Humble Gaussian – OK, not really pure linear algebra anymore, but quite enlightening. (12 pages)

After studying for this last stats/ML midterm, I’ve now printed them out and stuck them in a binder. A poor man’s linear algebra textbook.

I’d love to learn of more or different stuff out there. (There are always the appendixes of linear algebra reviews in Hastie et al. ESL and Boyd+Vandenberghe CvxOpt, but I’ve always found them a little too small for usefulness+understanding.)

Update May 2015: tweaked creditation for the CS229/CMU/Kolter review, fixed some dead links.

I *loved* EE263, which has a course textbook (not a real book, but rather the sort of thing that is printed at Kinko’s). But perhaps you have equivalent or more advanced material in whatever else you have from Boyd.

You mean this? http://www.stanford.edu/class/ee263/support.html

I like “crimes against matrices.”

This definitely looks like more of the right sort of thing than his convex optimization book/course…

Sam Roweis (RIP) had some useful stuff:

http://www.cs.nyu.edu/~roweis/notes.html

I liked Trefethen and Bau’s Numerical Linear Algebra. I used it for Math 104 at Stanford, though we only covered part of the book. Doesn’t have everything, but most of what seems to matter for statistics. I like the treatment of the SVD and QR factorization.

The first 5 lectures are here for sampling: http://www.amath.washington.edu/courses/584-autumn-2007/handouts/

I’ve gotten a lot of mileage out of Golub and Van Loan (mostly the perturbation results). I’ve heard good things about Horn and Johnson.

You could do a lot worse than Gilbert Strang’s classic text. His lectures from 1999 are online, too:

http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/video-lectures/

I have a friend who taught from Trefethen and Bau III last semester. He thought the book was a mixed blessing, being equally full of insight and typos.

Strang has multiple votes.

I’ve got Strang’s text also. I find it a bit too quirky and conversational to be ideal for reference. But it does have a broader range than Trefethen & Bau.

Would be interested to know what some of the typos in T&B are… I don’t doubt there are, but would like to fix them in my copy. I can’t find any errata online.

I like Carl Meyer’s book: Matrix Analysis and Applied Linear Algebra