Pyaudio audio playback is choppy

I’m working on playing and recording audio simultaneously in PyAudio and I’ve gotten it to work but the audio that is being played back to the user is very choppy. Also, I’m trying to delay the audio back to them at an increasing rate for this project I’m working on so I feel like that might also be effecting the audio This is what I have for defining the two audio streams:

streamIn = device.open(
    format=paFloat32,
    channels=self._CHANNELS,
    rate=self._RATE,
    input=True,
    frames_per_buffer=bufferSize
)

streamOut = device.open(
    format=paFloat32,
    channels=self._CHANNELS,
    rate=self._RATE,
    output=True,
    frames_per_buffer=bufferSize
)

‘bufferSize’ is a value specified by the user that represents the delay they want. So a buffersize of 44100 will represent 1000ms (1 second) of delay back to the user. This is how I play back the audio simultaneously (also where I think something is messing up):

while self._streamIn.is_active():
    start = clock()
    self._bufferSize += 44
    if self._bufferSize > 44100: self._bufferSize = 44100
    audioData = self._streamIn.read(self._bufferSize)
    self._streamOut.write(audioData)
    actualDelay = (self._bufferSize / 44) / 1000
    print(actualDelay)

I have the ‘self._bufferSize’ increasing by 44 every iteration which is about 1ms. When the delay gets to about 200ms the while loop I guess “slows down” and the audio the user hears gets really choppy and unrecognizable. I’m using PyAudio, does anyone know if this library would be suitable for this or if I should look into another one?

Quick note, I used code from this repo: https://github.com/snovvcrash/daf-generator just to give him credit for what I am trying to do