“I kind of resent you, but I’ll be damned if I’ll have it said that I forget peoples’ birthdays”?

Or perhaps, “I remembered your birthday, but I wouldn’t go as far as to say I hope it’s happy” would do.