While playing as Aragon I lost Sardinia (lower of the two islands west of Italy) to Sicily. A watchtower I had built in the lower right corner is still operating for me despite the fact that I do not control the island or have any units/specialists present there. It even says "Sicilian Watchtower (At War)" when mouse is over it.

Odd huh ?