The room you are looking at - now lined with books - was once filled with wool. People called the building, built in 1300, 'The Woolhouse' or 'Town Cellars'.
The export of English wool to France was big business in the Middle Ages, making enough money to build cathedrals. Poole was one of a handful of English towns allowed to export wool and this building was a kind of warehouse.
In the late 1700s a new street was laid that cut through the building. But it's thought that it might have been altered before. In 1405, a French raid ravaged Poole as revenge for the trouble cause by The Poole Pirate - Harry Paye - and damaged many buildings.