My biggest fear may seem rather insignificant compared to the fears of others, but it is something I think about regularly.
Sometimes, rarely but sometimes, I feel dirty and guilty when I know I haven't done anything wrong. For example, when I walk into a store and I am the only one getting followed. I hate it when people look askance at me because I am male, black or at one time, young. The worst is when you can taste the fear that a woman has whenever I happen to be walking behind them, or towards them. Many times (and there are many of you who probably know what I am talking about) they go as far as to walk across to the other side of the street. I even rehearse how I see a situation playing out if she decides to enlist the aid of a store owner, a passerby or even a police officer. "But I really was going in the same direction! But I live over there!" Sometimes when I pass them I think to myself, "See? I am fucking harmeless! I'm just a guy walking down the street that wants nothing to do with you!"
I wonder how many men suffer violence at the hands of anyone. It seems to me that as a black man in an urban centre, even (or especially, depending on your point of view) Montreal, I should worry about violence at least as much or more than any woman. I'd like to know how many of these men "had it coming", that is, were involved in illegal and dangerous activities which led to violence.
I suppose that being a white woman and being a black man are both relatively dangerous depending on where you are. Different people have different risks. I wouldn't last in many urban areas of the US. The stats for blacks are rather sad in many aspects of life. There are some places where kids dream of turning 18. Perfectly healthy kids. It's hard to believe. Probably as hard to believe that one in four women are victims of (sexual) violence at the hands of a man.
What this means is that I have to believe that men are actually that dangerous, which is hard, because on a global scale, it means that I am the enemy. I cause this pain, even if I had nothing to do with any particular incident. It means that it is reasonable to assume that I am a rapist, murderer or pretty much anything. It IS me, no matter how you look at it. It explains my biggest fear, being locked up or killed because I am male and/or black. Because I "fit the description". So far I have managed to keep out of trouble that I don't deserve. I wonder if that is true in a country where one in eight black men my age are in prison.
But I suppose that's another discussion.

I think one contributing factor to the prevalence of violence against women is the accompanying silence. So many women don't speak out about being attacked or abused. Statistics can only account for the women that do speak out.
The same could be said about any oppressed or abused group. Awareness is usually a good first step in stemming and eventually reversing certain abusive behaviours.
It's a tough situation. I too have felt the discomfort of knowing that the woman walking in front of me is worried that I might be following her. Of course her discomfort is greater than mine, but of a different sort.
In my case I try to own the situation by being aware of what I'm doing and what its effect might be. Specifically, I simply don't walk behind women at night. *I* will cross the street so she doesn't have to.
Some people might argue that I shouldn't have to do that, and it's true -- I shouldn't have to. And in fact I don't have to. I choose to because it makes both of us feel better.
Of course that's just the tip of this iceburg. With a lot of the stuff you mention there's really nothing you can do except to try to understand it as statistics and prejudice and not something about you as an individual. Ha ha ha! There you go, I've fixed everything! :-/