Sounds like your rear wheel is pointing left....or your saddlebags are not balanced....or you're sitting off center.
I align mine with a string on both sides of the rear tire running forward. Then measure to see if the front tire is centered between the two strings. Once you get it aligned, you don't have to unscrew the axle adjusters next time you remove the rear wheel. You can still get the axle in.