Probably a fault or air leak with the Servo Unit causing this, check all hoses for leaks and main unit diaphram inside has not split.
https://www.realoem.com/bmw/enUS/showparts?id=BT32-EUR-04-2003-E85-BMW-Z4_25i&diagId=34_1318
Diagram for parts on a 2003 2.5, put your last 7 letters and digits of Vin number in the home page to bring back parts for your actual car.