The location they filmed Sand Dollar Cafe is actually Carson's Store in Noank, CT.
Some of the same locations can be seen in Mystic Pizza (1988), including the shack that serves as Brody's bait shop.
The source novel was set in North Carolina, whereas this movie is set in Connecticut.