mine said the same.
she also said "inpatient is for people who are really sick, you're not there yet" when i disclosed that i was planning on going into inpatient care. i was admitted that night. months later, i still have trouble trusting that my providers actually believe me when i describe my symptoms.
a lot of the time, primary care doctors don't know jack shit about anything that isn't the common medical experience. my therapist, the group facility where i'm currently receiving services, and everyone in the er believed me when i told them i was in a dire situation and needed help. but my primary care doctor said two little sentences that have wormed their way into my head - and that's what makes it so sinister! the person who SHOULD know these things, who i trust to manage my health, is the exact person who minimises my experiences and does lasting damage because of assumptions she's made.
my occupational therapist, who treats mainly patients with chronic and/or invisible illness (and who is thus, in my opinion, far more qualified to say so) highly supports me using a wheelchair for my fatigue, dizziness, weakness and pain. she also supports me getting stronger and exercising - the thing my primary care doctor was likely trying to get at - but knows enough about it to understand why it's helpful and support me in my choice to use the aid.
tl;dr... get a better doctor if you can, lol. it's ironically hard to find doctors that are properly supportive of invisible illness, but they do exist (and i found one for myself after ruminating on my experience!) if you aren't able to switch, just know that while the intent was probably good, the effect is planting that seed of doubt, and that much is undeniably sinister. a physical or occupational therapist who specialises in invisible illness will be much more equipped to advise you on ambulatory wheelchair use and the effects on your health that can come with it. and they will likely be far more supportive and understanding while doing so ;;P