FS-7500: clean up logic of encoder flow and set rules on return values and tweak some codec settings on vp8