By what mechanism will people who didn't consent come into contact with LSD? As residue accidentally transferred by physical contact? By that logic peanuts and other tree nuts should be illegal because trances of its oils can kill those who are sensitive, unlike LSD which is pharmacologically safe. Additionally, while there are many examples of trace residues of tree nuts triggering anaphylactic shock in those not directly handling the nuts, I haven't heard of a single person accidentally ingesting LSD that wasn't willfully handling the drug (which implies consent to any accidental ingestion in my book).
A friend of mine was dosed without her knowledge or consent at a festival. She thinks it was from a cup of punch she was given, input to this day can’t be sure of the route of ingestion. She was familiar with LSD and realized what was happening, but was not prepared for it and had a rough time.
It’s completely irresponsible to give someone a drug without consent, outside of some forms of medical intervention, regardless of your intentions.