Modern vegetarianism got its start well before factory farming and other such practices. There has always been a vein of additional cruelty to animal agriculture (no captor is perfect, and there is little oversight, just as in the case of human slaves where there were better and worse situations). Beyond that, however, there is some intrinsic philosophical consideration to be give to the unnecessary taking of a life in and of itself.
Vegetarianism in the Western world developed as people became enlightened to the ways of other cultures (such as to the fact that some did not kill animals) and philosophy, and realized that the long supposed necessity of meat was dubious.
Once we (in a general sense, as Western culture) realized it was unnecessary, and quite crucially had the farming infrastructure and nutritional science to avoid it, it became a choice, with one side obviously taking what you could call the natural moral high ground.
Now, as to how moral or immoral it was (is it like stepping on somebody’s foot and not apologizing, or like genocide?) that has been and remains a question people debate to this day (even between fellow vegans).
It’s easy to agree that it is immoral (even many avid meat-eaters will agree to that), but quantifying that immorality is a very different matter.
Take a smoker- wherein smoking is immoral, because it harms your health and affects your loved ones, stresses our medical establishments with elected disease, and harms others who might catch wind of it in a more direct way as well- who even while understanding those points will continue to smoke.
Certainly, then, a non-smoking vegan is doing better than either the smoking vegan or the non-smoking meat-eater, but which of the latter two is falling behind, and by how much, or are they just about neck-and-neck?
It all comes down to quantification, which is at the forefront of moral philosophy- and is necessary should it hope to be a respectable science- as well as neurology and the scientific study of consciousness and the mind.
Of course, that’s not something I can go into at length here. Many opinions on many sides of that debate.