Maybe instead of having the conversation in person you can write it as a letter or email? It may be easier. Of course you’re still going to have that fear, but to be honest if he can’t handle listening to what you have to say, or if he gets defensive by it that’s not a reflection of you but him. In the grand scheme of things it’s a relatively minor conversation to have. Compared to the bigger ones that will happen if you stay tougher.
I remember having a similar thought process back with Chris. I wasn’t happy but at the same time I knew I would be unhappy single. At least I had something no matter how little it was. But It came down to this: as sucky as it would be to be single again, at least I had a chance to meet someone better. Even if it was a 1% chance. Where as there was 0% chance if I stayed. Or as my grandad used to say, 1 slice of bread is better than having no bread at all.
Don’t settle. If you do you’ll be kicking yourself in the ass later when you realize how much precious time you wasted.