"Never mind with you, you don't need to apologize to me"
it is not your business ,you don't have to apologize to me
It is none of your business.You dont have to say sorry.
you has nothing to do with it, no nee to apologize.
That's ok with you, you don't need to apologize to me
none of your business, do not appologize