In a 2017 interview with PunkNews, Jello Biafra touched on the release of Give Me Convenience or Give Me Death:
After we split, I thought even though it’s hard to get 7-inches stocked, people who only have the 12-inches have never heard songs like “Too Drunk to Fuck” or the b-sides like “In sight” or “The man with the dogs.” I put together Give Me Convenience or Give Me Death, the outtakes and singles album, that serves as a quasi-greatest hits album.