I agree, I stopped using toner a while ago and my skin has improved since then. I think the less you do to aggravate your skin the better!Contrary to popular belief, toner is not necessary most of the time. After washing your face, rinse with water and moisturise. Witch hazel is an irritant, so you might want to avoid that.
As Mari said though, toning does remove minute traces of product and dirt, but it also removes your natural oil barrier, which in the end is worse overall compared to leaving the face alone after washing.