Yes you are unequivocally right. but unfortunately I think we all kid ourselves about the Swiss Made on the dial for most of our watches.
You see, if you're looking at a watch that costs less than $5k (so that encompasses most of the mid-tier and affordable Swiss watches), they do incorporate many foreign parts, this has been known for a long time.
Hence for most if not all of the mid-tier and affordable Swiss watches we see in stores and on Drop, it is the high cost of the Swiss labor that allows them to put a Swiss Made on the dial, as the cost of the labor is greatly superior to the cost of the (finished or raw) parts that will have been sourced abroad. So these brands will all have many parts from abroad.
Some brands have less foreign parts (more expensive brands, think Tissot, Longines, Beaume & Mercier, etc.) and some will have more foreign parts (think more affordable brands, Steinhart, Glycine, Oris, Anonimo, etc.).
That's why I basically sold the many, many watches that I had collected in the past two decades, and stayed with Rolex, Tudor and Grand Seiko. Three brands that use mostly (if not all) in-house parts.