Struggling with a New Favorite Person

I'm having issues with a new Favorite Person. I have not had one since December 2020. and isolating due to being immunocompromised probably protected me from new connections and attachments like this. the issue is, he is moving. he's a regular at my work where I'm a server at an alehouse. we started hanging out and then started a sexual relationship. his house is on the market and he's moving out of state, he immediately started before anything happened that he didn't want a relationship. I do know he likes me. he's been distant lately and not as responsive as my ailment would like. in the last week, he has been barely giving me any attention. almost avoiding seeing me. two days ago he asked me to get food with him. he says we're good. and he still wants to hang out and that we started hanging out a lot when he wasn't working much and now he's working more. I asked him to hang out tomorrow or tonight. and he keeps saying he wants to see what his other options are first, which makes me feel like I'm not important. it's triggering me bad. it's all I think about. he knows about my BPD. says i go from 0-100 quickly. I explained that I see things in black and white and everything is one extreme or another. and he tries to understand. I explain communication is important to me. but he's being the opposite of that right now. should I just build a wall now? if so, how do I do that without my mental health completely deteriorating? I know I'm not his girlfriend. I know I'm just a friend he occasionally hooks up with. I'm putting a lot on him, especially in a stressful time for him and I feel bad. any advice?