Sounds rough. My fiancé does security, and from what I've gathered from him, the best time for security to get involved is at the design stage. They look over the proposal, give their input, and then nobody's surprised at release time, and teams can follow agile practices. Obviously there's still a review of the final product, but that can be done asynchronously after the fact to confirm that best-practices were followed.

Easy to say, hard to put into practice. Certainly depends on the kind of service your business provides.