While it is somewhat out of the way, Kampot is still rather well-connected to other parts of Cambodia. You can easily get there from Phnom Penh and Sihanoukville.
From Phnom Penh, there is regular bus service to Kampot. The cost of the bus trip is usually around US$5 and takes around three hours. A quicker and more flexible option is to use a private car transfer, which will run you US$25 to US$35 (or more, if you arrange it through a hotel that takes a cut). From Kep, you can catch a motodup or taxi. The fare is negotiable, around US$5 for a motodup or $10 for tuk-tuk.
From Sihanoukville, there is regular bus service for the two hour trip, or your can use a taxi service for around US$25.