feat: Implement emo_alpha scaling of emotion vectors and emotion text

- Added support for `emo_alpha` scaling of emotion vectors and emotion text inputs.

- This is a major new feature, which now allows for much more natural speech generation by lowering the influence of the emotion vector/text control modes.

- It is particularly useful for the "emotion text description" control mode, where a strength of 0.6 or lower is useful to get much more natural speech. Before this feature, it was not possible to make natural speech with that mode, because QwenEmotion assigns emotion scores to the text from 0.0-1.0, and that score was used directly as an emotion vector. This meant that the text mode always used very high strengths. Now, the user can adjust the strength of the emotions to get very natural results.

- Refactored `IndexTTS2.infer()` variable initialization logic to avoid repetition and ensure cleaner code paths.
This commit is contained in:
Arcitec 2025-09-11 04:07:58 +02:00
parent 555e146fb4
commit 9668064377

View File

@ -301,34 +301,42 @@ class IndexTTS2:
print(">> starting inference...")
self._set_gr_progress(0, "starting inference...")
if verbose:
print(f"origin text:{text}, spk_audio_prompt:{spk_audio_prompt},"
f" emo_audio_prompt:{emo_audio_prompt}, emo_alpha:{emo_alpha}, "
print(f"origin text:{text}, spk_audio_prompt:{spk_audio_prompt}, "
f"emo_audio_prompt:{emo_audio_prompt}, emo_alpha:{emo_alpha}, "
f"emo_vector:{emo_vector}, use_emo_text:{use_emo_text}, "
f"emo_text:{emo_text}")
start_time = time.perf_counter()
if use_emo_text:
if use_emo_text or emo_vector is not None:
# we're using a text or emotion vector guidance; so we must remove
# "emotion reference voice", to ensure we use correct emotion mixing!
emo_audio_prompt = None
emo_alpha = 1.0
# assert emo_audio_prompt is None
# assert emo_alpha == 1.0
if use_emo_text:
# automatically generate emotion vectors from text prompt
if emo_text is None:
emo_text = text
emo_text = text # use main text prompt
emo_dict = self.qwen_emo.inference(emo_text)
print(emo_dict)
print(f"detected emotion vectors from text: {emo_dict}")
# convert ordered dict to list of vectors; the order is VERY important!
emo_vector = list(emo_dict.values())
if emo_vector is not None:
emo_audio_prompt = None
emo_alpha = 1.0
# assert emo_audio_prompt is None
# assert emo_alpha == 1.0
# we have emotion vectors; they can't be blended via alpha mixing
# in the main inference process later, so we must pre-calculate
# their new strengths here based on the alpha instead!
emo_vector_scale = max(0.0, min(1.0, emo_alpha))
if emo_vector_scale != 1.0:
# scale each vector and truncate to 4 decimals (for nicer printing)
emo_vector = [int(x * emo_vector_scale * 10000) / 10000 for x in emo_vector]
print(f"scaled emotion vectors to {emo_vector_scale}x: {emo_vector}")
if emo_audio_prompt is None:
# we are not using any external "emotion reference voice"; use
# speaker's voice as the main emotion reference audio.
emo_audio_prompt = spk_audio_prompt
# must always use alpha=1.0 when we don't have an external reference voice
emo_alpha = 1.0
# assert emo_alpha == 1.0
# 如果参考音频改变了,才需要重新生成, 提升速度
if self.cache_spk_cond is None or self.cache_spk_audio_prompt != spk_audio_prompt: