Books: There are many technical books, some are so complex in their wordings that you'd feel like giving up even before you start.
The best book I've found that's short and sweet (but requires a little bit of math background) is this one:
Coursera has an Audio Engineering course which the professor explains in really crisp detail. It's from University of Rochester.
The course title is: "Fundamentals of Audio and Music Engineering: Part 1 Musical Sound & Electronics"
I audited the course and didn't pay any money. There's not much difference unless you want a certificate (and of course want to support Coursera too).
I like Kirby meets audio. He does some really nice stuff with speaker boxes. He mostly sells kits for his speakers, but I don't recommend them. They are from parts express mostly.
I LOVE John. He reviews most of the cheap amplifiers from China and teaches you what's wrong with them electronically and how to fix them. HE's very knowledgeable as well.
Where to buy parts:
If you're buying speakers, AliExpress usually sells good quality speakers for cheap. You may also want to search for the same model (sometimes under different brand names) on eBay as eBay can be cheap sometimes as well.
Contrary to popular belief, most speakers are actually made in China. Simply because China possesses most of earth's natural magnetic ore required for making speakers. That and Neodymium too. For example, look at the quality of this Aluminium cast speaker: https://www.instagram.com/p/BjZN5XKggOt/ . It's definitely hundreds of notches higher quality than what you can buy commercially from say, Sony or Philips in the consumer market. And it also costs really less.
Nanjing in China has most of these speaker factories and I visited a few last year. If you are able to make a trip to China like I did, it's best to form a personal relationship with someone there and you can order directly through them. The price difference between AliExpress/eBay and these factories direct can be as huge as $50-100 depending on the model and size of the speaker.
For amplifiers, I recommend eBay or AliExpress. Both carry decent models.
If you'd like to build your own, hands down the best model to get started with and easiest to hack one in a weekend is TDA 7297 by ST MicroElectronics. The good thing is, you can even power it with your USB! And the sound quality is REALLY good. It's the same IC used by a lot of companies (Creative for example) and resold to you for $700 (Creative actually has a model that uses this IC for this price, I can't remember which one). Many companies like Bose also use similar models from ST as well. That's when you realize stuff you buy from the consumer market is actually something you could have built yourself too, with a little more effort :)
Some other popular amplifier models (plug and play) are chips based on TPA 3116D2. They come in different configurations 2.1, 4 channels or good ole' stereo. Pick the ones you like.
Finally, don't be intimidated by all the complex terminology, sometimes you can just start with what you like or find combinations on the internet that are popular. Here's my earliest build: It's a mix of art + audio (attempt to): https://www.instagram.com/p/Bet9CA-FxHs/ . But even after so many years, I'm still proud of it :)
Hope this helps :)