The lugs could probably be glued or araldited back on if they broke, I think 2 of mine just cracked a bit.
The next bit holding the connector also can be prised off but all I can remember is I left all the wiring attached and hid it under the console.
I think I read that part of the wiring is needed or your phone wont work.