Cool, I guess this is sort a Farnsworth-Munsell equivalent.
Just to address this problem in isolation first, I think you are using code from this topic:
If you follow that thread down a little further you’ll find that person had exactly the same issue and that some more complicated code was suggested in this post to fix the issue of keeping a given stimulus as the selected one during a drag:
The issue of shuffling stimuli aside is another layer of complication. Let’s make sure you have the dynamic dragging solved first.