In order to avoid an IRS penalty, you must be 59 1/2 or older to withdraw funds. To avoid a surrender charge from COF, review your contract. All surrender charges are listed there.