On Liberty by John Stuart Mill Chapter 4 Page 14

circumstances render them at the same time duties to others. The term duty to oneself, when it means anything more than prudence, means self-respect or self-development; and for none of these is any one accountable to his fellow-creatures, because for none of them is it for the good of mankind that he be held accountable to them.

The distinction between the loss of consideration which a person may rightly incur by defect of prudence or of personal dignity, and the reprobation which is due to him for an offence against the rights of others, is not a merely nominal distinction. It makes a vast difference both in our feelings and in our conduct towards him, whether he displeases us in things in which we think we have a right to control him, or in things in which we know that we have not.