Is there a knack of putting it back in the clips,it should go back easy,but as the bishop said the last bit is proving troublesome .It came out easy enough.
If its the large part around the gearstick you mean I have had ours off a few times and never had a problem.
Maybe there is something out of place underneath it.