If women pass the physical, they should be let in. The subject has been raked over in previous posts. Its clear very few women will qualify for special forces but for many combat roles there will be a cohort of suitable physical specimens.
Women will experience harassment as surely as night follows day. Bands of young men in a military (or any social environment) placed under stress and bastardised by militarisation will practice exclusion, violence and other "bullying" behaviours. This actually seems to makes better soldiers, the survivors cohere into tighter teams, individuality is supressed in favour of predictable responses etc.
The elite sports comparison is inapposite, we need warm bodies on the ground and we've been excluding over half the population because of Victorian anxiety about "the ladies".
As for rape concerns, a mate of mine in the ADF was given a briefing on rape when he arrived in Iraq. I will spoiler it as its a bit rough.
Spoiler Alert, click show to read:
Rape is a part of war and happens to men and women. US forces at Abu Graib engaged in sexualised humiliation of prisoners, and I would guess the US defence forces would be one of the more humane and disciplined forces in the world.
Yes women soldiers will be raped as male soldiers are by colleagues and more often the enemy. Some will be shot, some will be bullied (probably at a higher rate than male colleagues as they make up a smaller group, and for reasons of social inequality in the communities recruits are drawn from-"make me a sammich" etc) and many will die. That's what happens to soldiers.
If women are an equal part of your political community they have equal obligations (and should have equal opportunities) to defend it, subject to suitability to the role.