If you owe the IRS taxes, you will have to pay a failure-to-file penalty, a failure-to-pay penalty, and interest on the amount owed. All three are assessed based on the amount of your tax liability and on how long your taxes have remained unpaid.