The side panels on the Ascot are held on by pegs in three grommets.  This is a great system as long as the grommets are in good shape, but when they get worn it can mean that your side panels will part with the bike on the road.  You probably won't even know they're gone until they're ground to dust under following traffic.
The side panels are EXPENSIVE!  The three grommets that hold them in place are CHEAP (relatively).  Save yourself a world of trouble, and a UGLY bike by replacing the grommets when they get worn.  Part #  83551-300-000

One other thing you can do is to add a safety wire to each side panel.  Maybe add a wire a foot long to the front mount of the side panel and tie it to the frame.  That way, you have a chance of stopping while it's still flapping around on the bike!