Not always true. In most cases it's not even remotely true.
The first thing you have to understand is that there are different kinds of audits. A mail audit is simply the IRS trying to verify a specific line item, and it involves you mailing in all of the support documents for that particular figure [source:Colorado Bar Association].
If you can prove the number is correct, then it should lead to a quick and hassle-free conclusion.
Mail correspondence is the most common form of auditing these days, with about 76 percent of audits occurring in this way [source:Ellis]. The other form is an in-person audit where an IRS agent will request an appointment with you to provide certain financial information
That being said, whether or not the auditing process is a nightmare also has a lot to do with you. If you're organized with your financial records, it will be a lot less hectic than if you don't know where anything is.