Someone once said " You don't owe them an apology, if they are bothered by the boundaries you've made"