Fiverr can introduce a rating system which collects all ratings and reviews but does not show them by 5 stars, 4star. 3,2 or 1 star.
Let it be as most recent /oldest/most favourable/most critical sorting only and let there be an average rating to the seller profile and seller gig. The seller won't get to know which buyer left which no. of stars and let buyer also not know the star rating left by seller for him/her.
As sellers with more orders won't get to know who left which rating, they can rate buyers more genuinely. This can obviously be apparent as a new seller starts his profile, but for gigs with more orders, a seller may not know exact star rating left by a buyer as average will be around 4.9.
This will reduce the case of retaliation by sellers and buyers.