I find it hard to believe the batteries can move to the extent of bending the driftshaft. Mine are really snug under the strap.
The only way you'd be able to change batteries without removing the spur (or lifting it up like I do without complete removal) is to mod the side pod ilke they did in the York 24hr race so the batteries can be slid in and out.