It’s nice to see that you actually are able to use smaller latencies with this method!
I saw in your implementation that you are reading from a sound file within the callback. This might seem to work well if the OS buffers nicely, but at some point reading from the file might take more time than is available in the time frame of the callback. I think you should implement some kind of buffering to avoid underflows.
Speaking of which, I think it is really important to always check the status argument of the callback function which will tell you if an xrun occurred. This should never happen during an actual experiment, because it might result in an audible click!
You mentioned “streams”; are you talking about PortAudio streams?
If yes, you should note that multiple streams are not officially supported by PortAudio.
They might work for some host APIs on some systems, but if you want your code to be platform-independent, you should limit yourself to a single stream.