Understanding Cryptography: A Textbook for Students and Practitioners: Great resource for understanding the math and logic involved in modern crypto. I read through and took notes, but need to go back and do many of the exercises. It professes to not be a cryptanalysis book, but does a good job of covering the weaknesses and common attacks for many schemes.
The Code Book: The Science of Secrecy from Ancient Egypt to Quantum Cryptography: Entertaining read. I would consider recommending to non-computer friends, but still a good survey of the field.
Cryptanalysis: A Study of Ciphers and Their Solution: Recommended in the Bruce Schneier syllabus. Very dense and seems somewhat dated. Slow progress.
The Codebreakers: The Comprehensive History of Secret Communication from Ancient Times to the Internet
picoCTF Crypto Challenges: These seem mainly focused on either old attacks (frequency analysis, known plaintext, etc) or implementation mistakes (OTP reuse), but still fun.