That’s poor form - frustrating for you and a waste of an appointment, so adding to the protracted waiting times for OPD clinics.
If your appointment is with a spine surgeon or physio, I expect that they would want to examine you out of the brace and consider repeat X-rays, maybe including flexion and extension views before letting you go without it. They might put you in a soft collar for a few more weeks, but with bathroom privileges.
Probably best you don’t just rip it off yourself and go home!