I had the handle fall off a while ago and found the hinge pin inside the handle had worked loose, it looked like the thin scroll hinge pin was missing. If like me you cant seem to find the hinge pin, it's thin and has slipped back inside the hinge tube so is hidden from view. It's a fiddle to re-assemble with the small spring but is a zero cost repair once it's all put back where it should be and has been OK for the last 5 years.
http://bmwfans.info/parts-catalog/E85/Europe/Z4_2.5i-M54/R-N/apr2004/browse/vehicle_trim/glove_box