venting about my partners family in my partner

Me and my partner have been dating for nearly a month and what a fucking month it’s been already.
my partner had a baby with another women before meeting me and they haven’t been together for a year and something months but from what he has told me and his family has told me. she fucking crazy like bat shit crazy and that’s fine with me i have no interest in ever talking to her for anything other then the kid but she went to his house looking for me because she’s pissed off that he moved on even though she is the one that ended their relationship. she saw a post i had made on facebook and i tagged my partner in it and she was not happy to say the least, so with all the baby mumma drama, his family is very protective and doesn’t believe that he can find and be with someone that isn’t fucking bat shit crazy and i met his grandparents and mother last week. his grandmother has always been nice to me and has always loved talking to me even hugged me when i got there however after we left and went to the gym his grandparents have been very off with him about me- won’t let him come here or anything and i mean he is sick but its not contagious and he feels fine and is on medication but she’s made snarky comments thinking that my partner wouldn’t tell me and would keep a secret from me and it’s made me not want to be around any of them and i’ve still got alot of his family to meet yet and im scared it’s going to be the same thing with them too.
Now i don’t know what to do because eventually i’ll have to meet them but i’m trying to put it off the best that i can and he won’t let me do it forever.