Anxious around my friend's brothers

Does anyone else get anxious around certain type of men? Recently I realized I got anxious around my best friend's brothers. I brought my kids with me to spend an hour together with my friend and she was okay with it. She share a home with her brother. They all pay their parents rent and she is usually only in her bedroom not the main part of the house. She said it was okay to bring my kids since no one could watch them. She wanted to use the livingroom since it is more room for all of us. She told her brothers that I was bringing my kids. I saw how they reacted when she told them again when I got there. I saw how they reacted and I suddenly got into a anxiety attack. I felt very uncomfortable and I had to go into her room with my kids and her to breath. I was still anxious for about 15 mins. I didn't have my anxiety meds with me. I was fine until I saw their reaction and I had an anxiety attack. I was rubbing my hands and looking around and hovering over my kids to make sure they stay near me. This isn't the first time I seen them and each time which I bring my kids or not I get anxious and they make me uncomfortable. Is that normal? since I only have the reaction towards them or other guys that look similar to them. like I haven't even known them for more then a month and I still get anxious I don't talk to them either or try to be friend them since I get anxious around them. They also seem a little rude.