For me a major element of the witch figure is that the witch tends to be very values-driven but that these values are not always apparent to other people. This creates an ambivalence in other people because it is difficult or impossible to read the witch's inner world and values.
Combine this with the simple fact that most witches are past masters at reading the situation around them and thus knowing what is coming next, and this is a figure almost guaranteed to cause discomfort in other people.
To take this one step on, it is only necessary to consider that the mere presence of the witch makes things happen. Now for example, I have a tendency to dig my heels in, and also one to shoot my mouth off. This means that I will perpetually be in situations where those actions are either necessities or else where I have to learn to deal with those tendencies and manage them properly.
Years of the witching have taught me that there is no point trying to avoid that. I am comfortable with this simple fact now - and surely a failure to agonise about 'why me' must be one of the most frightening things possible to the world around me. That said, the fact that the sort of situations in which I am required either to dig my heels in or shoot my mouth off, magically create themselves around me, is probably the most frightening aspect possible of the witch figure, for other people that is.
You see, it is not enough that people habitually project all their fears and insecurities onto the witch, it also happens that an encounter with the witch will involve lessons for other people. You can phrase it how you like, and I have heard it called correcting karma and all sorts of things, but this is the main purpose of the witch in the world: to do ones own work in this incarnation and enable or cause others to do theirs.
Muggles are often further surprised by the working out in the witch's life of the magical axiom that apparent polar opposites are connected to each other in such a way that they can easily become each other. For example my tendency to dig my heels in means that the lesson people can often have to learn from an encounter with me, is the apparently opposite one, that things have to come to a natural end and there is no point trying to spin them out further.
I suppose the point of these reflections is that the witch figure - and so the person of the witch - is one that is largely fated to cause discomfort in others. But that said, naturally the polar opposite is also the case. An encounter with the witch also causes laughter and pleasure because once the karma is dealt with another feature of the witch thing is that balance is to be sought. I know I keep banging on that both the black magicians and the white lighters are on a hiding to nothing, but it happens to be true! The comforting yet terrifying lesson from a witch encounter is that you can't expect life to be all good or bad, and in fact those divisions may well be the most illusory things there are.
Now if people would just stop blaming the witch for anything they don't like, life would be perfect...