For mainstream classical, haunt Half Price Books in the U-District (4709 Roosevelt Way NE) and on Capitol Hill (115 Belmont Ave E). My favorite online store, Berkshire Record Outlet (www.berkshirerecordoutlet.com), abounds with hundreds of well-described classical and avant cutouts starting at $1.99. You can also consult www.cdshopper.cjb.net, which consolidates coupons for amazon.com and other web-based retailers.
One tip: check the fine print on back. "Historical" (read: pre-1952 lo-fi) classical recordings aren't for everyone and sometimes deceptively appear repackaged. Most used-CD stores around town have a jazz section, but labels like Verve and Blue Note have refurbished and reissued seminal releases multiple times, so be careful. My Japanese import Birth of the Cool CD purchased in the early 1990s sounds like crap compared to the 2001 remastered "Rudy Van Gelder Edition." For specific discs (and smart recommendations), Bud's Jazz Records (102 S Jackson St) remains at the top of my list.
Hunting for the avant poses a real challenge, but anything weird usually gets misclassified except at specialty stores, so check the Clearance and (swallow your pride) New Age sections. (Last time I checked, Cellophane Square [4538 University Way NE] files lowercase sound artist Bernhard Günter under "Rock.") Wall of Sound (315 E Pine St) has an interesting used experimental section, as does Everyday Music (112 Broadway Ave E).