You perception is valid. Even if a person has light skin in the USA, they can be perceived as “not really White". This happens if one is bilingual or multilingual, has cultural influences from non-English speaking countries, or speaks with an accent. If you were out in public speaking Spanish, it would be obvious.