Whisper PlaygroundをLLM Broker対応にする

Whisper PlaygroundはZIKUU内で使われる YouTube動画の文字起こしアプリです。

Whisper自体はCPUを使った別ルートですが、外国語を日本語に翻訳する機能で、LLMに翻訳をさせています。
その部分をLLM Broker対応にしました。

この動画

https://www.youtube.com/watch?v=Lgj3OWgNjMA

を文字起こしした結果がこれ。

U.S. President Donald Trump's polling numbers slipped to some all-time lows this week. A CNN poll suggests Trump's overall approval rating has slumped to 34 percent, although his polling hasn't hit since the January 6th insurrection. Trump started out his term with a much higher net approval rating, but is now sitting in the negatives as the war with Iran drags on and gas prices stay high. His numbers continue to decline. With the U.S. midterms quickly approaching, can Donald Trump afford a dip in approval ratings? Ruth Gellnick is a polling editor with The New York Times. Ruth, thanks for doing this. I appreciate it. We've had a handful of surveys out of recent, sort of the last couple of weeks that are showing this slip in popularity for Trump. Just zoom out for us. And what do the numbers tell us about the broader picture here? Yeah, so in our polling average, we collect all polls that are conducted about Donald Trump's approval rating at The New York Times. And we saw sort of a slight dip in Trump's approval rating. He's down to about 37 percent in our average, which is a historic low for Donald Trump. It's as low as he's been in this term. He's been at 37 percent in his previous term, but that's the lowest we've seen in this term. But sort of zooming out, this is a small dip, but not a major dip. And it's sort of in line with where his approval rating was in April and May of this year. He had a slight uptick in June, but now it's kind of back down to that low that he's been over the summer. And can we safely say that's largely the Iran war and, by extension, the cost of gasoline? That's the one thing that really seems to move the needle, isn't it? Yeah, that's absolutely right. Americans are really focused on costs and how things impact them. And the war is very unpopular. And the sort of cost outcomes there, particularly around gas, is deeply unpopular. And that's really impacting Donald Trump. You know, Canadians, as I'm sure you know, have a sort of, you know, a very self-interested reason to be following all this, because we're trying to understand how Trump's popularity will either embolden or make him move on certain policies. And we saw this in the first term, like the first round of the Trump presidency. We saw he had a bunch of policies that weren't very popular, and his popularity dropped off about this point in the first term. And then we started to see some change in those policies. So just walk us through, how does this moment compare to that moment in the first term? Yeah, it's pretty similar. I mean, it's interesting zooming out even more before Donald Trump's first term. These kind of things like wars and economic impact used to move the needle 20, 30, 40 percentage points. That's not really the case anymore in America. Americans are pretty entrenched in their political tribes. So we're talking about things that move the needle 5, 10 percentage points. But this does sort of compare to Trump's first term. He does some unpopular things, his popularity dips, and then those recede from memory, or people just aren't paying attention in the first place, and his popularity starts to tick back up again. So I think it's sort of a really similar pattern that we've seen play out for Trump over that his last term and this term. You know, I look at the headline numbers, the 34%, the negative 20 net approval number is a big one. There's a YouGov poll this week that showed 72% of Americans say Trump's tariffs are increasing costs of stuff that they're buying. And then on these latest round of tariffs that Trump is threatening that would come into effect on August 19th, if they go ahead, 60% disapprove, 49% strongly disapprove. And well, I think that's probably good news for Canada. The question I have at the end of the day is, do we expect those kinds of numbers might actually move the policy needle even a little? No, we really don't. We saw this with with Trump and tariffs in the past. Tariffs do not tend to move the needle. Like I said, Americans are sort of focused on on their own costs and things that they understand. So they're focused on gas prices. They're focused on things they understand that are in front of them. And when we look at how people are assessing tariffs, they're not always thinking about the big picture, and they don't even fully understand them sometimes. Tariffs are kind of complicated concept. And we saw that when we looked at different ways that pollsters asked about tariffs that really impacted Americans responses, which tells me if it's very sensitive to a little wording difference, it means it's not a deeply held opinion. It means it's not something people understand and they feel really strongly about. And it doesn't really move the needle on how they feel about Trump. They may react to it in a poll and say, yeah, that doesn't sound great, but it's not moving the needle overall on how they feel about Trump's presidency. You know, the broad view, like the historic view, you've been analyzing polls for a while. In your experience, numbers like this, they look bad. But can a president and especially a president like Donald Trump, I guess those are two questions, is he different than everybody else and can a traditional president bounce back from numbers like this? A president can bounce back from numbers like this. Absolutely. And it's not that Donald Trump is necessarily different than other presidents. It's that this era in American politics is different, and it's been the case for the last 25 years. So this was the case under Biden as well. We've become more polarized. We've become more politically polarized in our sort of tribes. And so what you're seeing is the real number that matters here isn't necessarily Donald Trump's approval rating with all Americans. It's his approval rating within his party. It's his approval rating with Republicans. And there he's still doing very well. And so there's almost you don't need to bounce back from that. He's doing very well with Republicans. And so he's going to continue to do well. I think, you know, we haven't seen a lot that moves the needle with Republicans and really tanks his approval rating with them. And does it change that? I mean, in theory, at least he can't run for another term president. He's less beholden to the polls, at least in theory. Yeah, I mean, there's still a midterm election coming up this fall. And I think Donald Trump is, you know, seeing the consequences of endorsing different people in races. So I think his party, it still matters for his party. And you're seeing leaders in the Republican Party, you know, make clear how much it still matters. But at the same time, like I said, he hasn't lost Republicans, his approval rating with Republicans is 75, 80 percent. So he continues to do well with his base. He continues to do well with the people that sort of matter to him in the moment. I was just looking at the poly markets, so the prediction markets, which I'm sure don't land very well with the pollsters. But I was looking at those about the midterms. And they're pretty mixed in terms of can he take the Senate, probably take the House. What is the polling picture heading into that midterms, like less than 100 days away now? Yeah, final stretch. You know, the polls right now show Democrats with an edge. A lot of polls have had Democrats up seven, eight percentage points in some of those midterm head to heads. But Republicans also have a pretty good structural advantage. A lot of the Senate seats that are open, you know, favor Republicans generally. Obviously, we've heard a lot about redistricting and gerrymandering. So there are a lot of reasons that Republicans kind of go into this with a bit of a structural advantage. And Democrats are doing well in these polls. The question is, are they doing well enough to overcome some of those structural advantages? And just quickly, lastly on the on the midterms, it feels like the Democrats are kind of all over the place, but they have to fight so many different kinds of battles in so many different states. It feels like from the outside, at least the Republicans often fight one more coherent, you know, national battle than the Democrats do. Is that a fair assessment? Do you think? Yeah, it's hard to say. I think right now the challenge is, you know, Republicans are on offense and Democrat Republicans are on defense and Democrats are on offense. And so it's, you know, it's easier to defend your territory than to fight forward. And so I think Democrats are having that fight forward in different areas, which necessitates different kinds of fights. So, you know, it's hard to say if one is more unified than the other, it's just a different kind of fight. Indeed it is. Listen, it's been really great to get your insight on this. Appreciate you coming on the show. New York Times polling editor Ruth Galmick. Thanks for this. Thank you.

これを翻訳したのが上のスクリーンショットです。

CPUでWhisperを動かして、8分の動画から文字起こしするのに70秒。

翻訳が2分くらい。
生成速度は、RTX 2000 Adaで 65〜70トークン/秒くらい。
文章が長いので、小さなチャンクに分割して、それぞれにLLMで翻訳してから１本につなげています。

---

一応、Whisper Playgroundは、文字起こしした文章をNerveに送信する機能がありますが、それを受けるハンドラーがありません。作るとしたらCPSで文脈抽出するハンドラーです。

個人的には、物理、天文、数学などの解説動画から文脈束を蓄積したいと思いますが、YouTube動画には、脚色されているものが多いので、長文のわりには蓄積すべき部分が少ないことが多いです。

---

LLM Brokerを使う利点は、

• 用途ごとにプロフィールを作ることで、LLMへのパラメーターの既定値を設定できること
• バックエンドのLLM Providerをプリケーションから隠せること

など。

これがプロフィールの例です。

# CPSの文脈抽出/Fact BuilderのDimension抽出 
  text_extract:
    backend: llamacpp
    base_url: http://192.168.0.197:8081
    model: gpt-oss-20b-Q4_K_M
    temperature: 0.1
    timeout_sec: 360

# Chat用
  text_conversation:
    backend: llamacpp
    base_url: http://192.168.0.197:8081
    model: gpt-oss-20b-Q4_K_M
    temperature: 0.7
    timeout_sec: 360

# 翻訳用
  translation:
    backend: llamacpp
    base_url: http://192.168.0.197:8081
    model: gpt-oss-20b-Q4_K_M
    temperature: 0.0
    top_p: 0.9
    top_k: 40
    min_p: 0.05
    repeat_penalty: 1.15
    timeout_sec: 360

現時点では、LLM Brokerの機能は、

• llama.cppとOllamaのバックエンドに対応
• Stream出力対応
• Tool Call対応
• OpenAI API準拠のChatCompletions API (Open WebUIなどのツールから接続できる)
• OpenAI API準拠のResponses API (Codexから接続できる)

程度です。

これだけでもアプリケーション開発と運用がかなり楽になりました。

今後、Vision、キューイング、負荷分散、クラスタリングの機能を追加していきます。

これは、

• 速さを犠牲にしても処理を詰まらせたくない
• 用途ごとにGPUを使い分けたい
• 問題発生時にフォールバック経路を持ちたい
• 多くのリクエストを同時に処理したい

といった要求に応えるためです。