Galle Fort is one of the attractions that, in my opinion, is worth a visit in Sri Lanka. Moreover, getting there from many resort towns is not difficult. You can get there either by tuk-tuk or local buses.
I already posted a review about my trip to Sri Lanka here http://www.turpravda.ua/lk/unavatuna/Comaran_Beach_Hotel-h34589-r76587.html but I only wrote about the hotel and Unawatuna Bay.