Feeling Unsafe at Work: Dealing with Hazy Memories of an Ex

My employer is considering hiring my ex. I have expressed that I do not feel safe around this person (and my employer is being very understanding) but I am worried that my fear towards this person is unwarranted. I remember the genuine good times we had together pretty well, but the bad is all hazy. I cannot remember what in particular they said to make me so upset and it makes me question if my beliefs are warranted. I do somewhat remember this person kissing me without my consent, but feel guilty for being upset about that, as it could’ve been worse or I could have done something that made them think it was ok. How do you trust that your hazy memories are correct? Am I right for being scared of them? Or are my fears unwarranted because I cannot remember much of the bad, just the pain?