What would be the use case for more than two handles?
I would probably do this be adapting my interactive slider code rather than trying to use the slider component itself.
What should happen if someone clicks on the slider but not on a handle? I’m guessing this would work as a drag and drop where the y (or x) coordinate of the handle is fixed.
I should have been clearer - I would only ever require two handles.
As for your other question, ideally, the only interactive parts of the slider would be the two handles. Assuming a horizontal slider, I would just need the two handles to move freely on the X plane.
I would suggest using two sliders next to one another to give the appearance of one bar. Then use the difference in ratings to modify a range bar. Like this