First question is why? Don't get me wrong, Bury St Edmunds is lovely it's just quite small! Suffolk generally is lovely even with my massively biased view. You should check out the following depending on what you're into or after. All within and hour or two of your base:
Lavenham - old stuff, pubs (Swan is good), shops (old)
Kersey - postcard type village near to Lavenham. Stream in road with ducklings etc. Kersey Bell (pub)
Cambridge - only half hour away from Bury SE. Obviously lots to to.
Snape Maltings
Thorpness/Aldeburgh/Southwold/Walberswick etc etc etc
Snetterton Race Track