the only settings here are the selection of the virtual audio cables and the level settings:

The audio cable Line-2 was selected on the receiving side of the SDR console. This reception signal must therefore be selected as the audio input for the HSmodem. The output on Line-1 corresponds to the transmitter.

The volume should be in the middle. Under no circumstances should the yellow bar turn red, as the signal would then be overdriven or underdriven. The input level is set so that the spectrum is roughly in the green area.

This picture shows the active HSmodem if all settings are correct. With this rather sharp constellation diagram (8APSK 6000 bit / s) data can be transmitted without errors. With SDR consoles and Pluto, you should not be satisfied with less.