Do you still face this problem? Sounds like a saved session issue. Check if autosave is disabled:
$ xfconf-query --channel xfce4-session --property /general/AutoSave and make sure the checkbox to save that appears on logout dialog is also unticked. Then, clear out ~/.cache/sessions Wouldn't also hurt to see your ~/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-panel.xml
