Producing picture and sound from one sequence rather than making them separately and stitching them is where H3’s design starts.

Why This Matters

This is for people running an in-house GPU cluster who must decide whether to self-host a video generation model or call an API. The conclusion first: H3-Base is a 33B-parameter single-stream transformer, so on paper it resembles LLM serving, but the real bottleneck is not the weights, it is sequence length. A single 15-second 2K clip produces a sequence of over 325,000 tokens, and the initial release ships without the sparse attention implementation that would make it affordable.

Overview

MiniMax H3 is a general-purpose omni-modal generative system. It jointly understands multimodal context made of text, images, video, and audio, and generates video with stereo audio from it. Output specifications are 4 to 15 seconds, 24 FPS, 32 kHz stereo audio, with a default short side of 768 pixels. It supports aspect ratios from 21:9 through 9:16, and dialogue in 11 languages with stable support, including Arabic and Korean.

What separates it from earlier video generators is that there is no seam where the modalities were sewn together. Text, vision, and audio tokens do not sit in separate silos and merge at the end; they share one transformer stream. In the model card’s phrasing, neither the attention layers nor the FFN layers contain modality-specific structures, and modality-specific parameters live only in the input and output layers and the AdaLN branches.

The license is the MiniMax H3 Community License. It is not a standard open-source license like Apache or MIT but a bespoke community license, so read the clauses first if you are considering commercial distribution.

What the System Is

H3 is not one model but a system of three modules. This distinction is the first thing to know when planning self-hosting, because only the middle one is open source.

flowchart TB
    A["User input<br/>text · up to 9 images<br/>up to 3 video clips · up to 3 audio clips"] --> B
    B["H3-Context-IR<br/>instruction parsing · cross-modal association · temporal understanding<br/>not released, API only"] --> C
    C["Context Intermediate Representation<br/>structured intermediate form"] --> D
    D["H3-Base<br/>H3-Encoder + H3-VisualVAE + H3-AudioVAE<br/>+ H3-Omni-Transformer 33B<br/>open weights released"] --> E
    E["768p output<br/>video + 32 kHz stereo"] --> F
    F["H3-Regenerate-2K<br/>regenerates with the original context<br/>not released, API only"] --> G["Final 2K output"]
    E -.local deployment reaches this far.-> H["Self-hosting boundary"]

H3-Context-IR is a hosted preprocessing and orchestration system for free-form multimodal input. Because it relies on a multi-stage workflow and multiple hosted models and services, it was not part of this release. MiniMax provides an API that reproduces the official workflow, plus prompting guidance for building your own preprocessing system. But as the model card states explicitly, H3-Context-IR is critical to final output quality, so running H3-Base without it will not match the official demos.

H3-Regenerate-2K is also unreleased. The interesting part is that it is not a separate super-resolution module. It feeds the 768p result back into H3 along with the original multimodal context and regenerates at 2K. The advantage is that small text and fine detail, which conventional super-resolution has to guess at, can be recovered from the original context.

The middle piece, H3-Base, is what was released. Text is encoded by the H3-Encoder, which uses the full pretrained weights of Qwen3-VL-32B and passes hidden states from its 50th layer to the transformer. Visual inputs go through both the H3-Encoder and the H3-VisualVAE; audio goes through the H3-AudioVAE alone. The H3-Omni-Transformer then jointly predicts video and audio latents.

The VAE specifications matter for the arithmetic later. H3-VisualVAE is a temporally causal video autoencoder with 16x spatial compression, 4x temporal compression, and 24 latent channels. Patchification of 1x2x2 along time, height, and width is applied on top, so visual tokens entering the transformer have an effective spatial downsampling factor of 32x. The temporal factor stays at 4x. H3-AudioVAE uses the same encoder and decoder for the left and right channels while processing each independently, compressing 32 kHz audio into latent tokens at 40 Hz per channel.

Installation and Integration

Total weight first. We pulled the file manifest from the HuggingFace API and summed only the safetensors.

curl -s "https://huggingface.co/api/models/MiniMaxAI/MiniMax-H3?blobs=true" \
  | jq '[.siblings[] | select(.rfilename|endswith(".safetensors"))
         | {f:.rfilename, b:(.lfs.size // .size)}]'

The calculation script lives in the ThakiCloud repository. It sums bytes per module and derives sequence lengths in the same run.

.venv/bin/python scripts/experiments/minimax_h3_serving_budget.py

The minimal Diffusers path is in the model card.

pip install -U diffusers transformers accelerate

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "MiniMaxAI/MiniMax-H3", dtype=torch.bfloat16, device_map="cuda"
)

To be straightforward about it: we did not load H3-Base and run inference for this piece. We computed the resources it requires below, but we did not attach a node of that size to this task. So there are no numbers here on generation quality or measured latency. What there is comes entirely from published file sizes and published compression factors, derived deterministically.

Measured Results

The safetensors across the whole repository total 464 GiB. Do not read that as the required capacity, though, because the same weights are laid out under FL2VA, Ref2VA, and the repository root. Deploying one variant actually requires this much.

The 33.14B derived from bytes matches the 33B the model card states, which confirms the calculation path.

One sentence in the model card becomes important here. Of the transformer’s 33B, roughly 13B sits in AdaLN-related branches, and because AdaLN modulation outputs can be precomputed and cached, those parameters do not need to be loaded for inference-only deployment. The full weights were released to support downstream development including fine-tuning. So if you only plan to run inference, the transformer side drops to about 20B, or 37.5 GiB in bf16.

Left is per-module weights summed from the file manifest; right is sequence length derived from the VAE compression factors.

But the real problem is not the weights. It is the sequence. Applying the compression factors above:

Here is how to read it. A 15-second clip is 360 source frames, which 4x temporal compression turns into 90 latent frames. Taking 768p 16:9 as 1376 by 768, the effective 32x spatial compression gives 43 by 24, or 1,032 tokens per latent frame. Multiplied by 90 latent frames that is 92,880, plus 1,200 audio tokens. Going to 2K pushes tokens per latent frame to 3,600, for a total of 325,200.

Token count grew 3.5x, but full attention cost grows quadratically, roughly 12x. And here the model card states something important. Native sparse attention was introduced in the final stage of training to cut the cost of long sequences, but the initial open-source release provides inference with full attention only, with the sparse-attention implementation to be published separately later.

That is the single most important sentence for a self-hosting plan. The gap between what the official API produces and what you can produce locally with the weights you just downloaded is not only a quality gap but a compute-cost gap. Processing a 94,000-token sequence with full attention is a completely different budget from having the sparse implementation.

What This Means for ThakiCloud Products

ThakiCloud’s ai-platform schedules GPU workloads on K8s and Kueue and runs multi-tenant vLLM-based serving. A model like H3 makes several concrete demands of that structure.

First, the deployment unit changes. At 134 GiB, a single variant nearly fills one H200’s 141 GB with weights alone. Once you account for activations and attention workspace for a 94,000-token sequence, one card is not enough, and even dropping AdaLN for inference-only leaves little headroom. A realistic configuration is two or more H200s, and serving both the FL2VA and Ref2VA variants costs proportionally more. Expressed as a Kueue workload, this model is hard to place in the same queue as LLM serving pods. Per-request occupancy is long and the memory profile differs, so a separate resource flavor is the better split.

Second, queue design changes. LLM serving streams token by token, so requests pass through in short bursts, whereas video generation holds a GPU for a long stretch on a single request. On top of that, the 2K workflow adds a regeneration pass after the 768p generation. Since user-visible response time is the sum of both stages, the queueing policy should be designed around completion time rather than throughput.

Third, it fits on-premise demand well. Video generation material is often sensitive by nature. Raw footage, product imagery, and video containing in-house individuals are assets that are hard to send to an external API. Being able to run an open-weight model inside the customer’s boundary satisfies that requirement on its own. That said, with H3-Context-IR and H3-Regenerate-2K unreleased, a fully on-premise configuration is not currently possible. You must build your own preprocessing system following the prompting guidance, and 2K either goes through the API or you settle for 768p. Explaining that gap accurately to customers is the most honest thing to do at this stage.

There is one thing to add from the Paxis lens. What H3-Context-IR does, namely parsing free-form multimodal input, interpreting cross-modal relationships, and serializing them into a structured intermediate representation, is really an agent orchestration problem. Its being unreleased means, conversely, that the slot is open for you to fill. Using Paxis’s DAG multi-agent composition and policy gates, you can make the input refinement pipeline auditable and trace which prompt enrichment led to which output. For an organization that must answer for generated results, that traceability is a prerequisite ahead of image quality.

Limits and Counterarguments

The numbers here have clear boundaries. We did not load the model and produce video, so generation quality, real latency, and real peak memory are absent. Weight sizes come exactly from the file manifest, but activations and attention workspace vary greatly with implementation. The 134 GiB above is therefore a floor on required VRAM, not the actual requirement.

The sequence length calculation also rests on an assumption. We took 768p 16:9 as 1376 by 768 pixels, but the model card states only that the short side is 768, so the long side may differ slightly by implementation. The effective 32x spatial and 4x temporal compression are published values, however, so the order of magnitude does not move.

The license deserves a note as well. The MiniMax H3 Community License is not a standard open-source license and includes use restrictions. The model card states that user-submitted text, images, and videos, as well as enhanced prompts, are subject to automated moderation, and content suspected of being unlawful, pornographic, or infringing third-party rights may be blocked. It also states explicitly that these guardrails do not replace the licensee’s obligations. If you are planning a commercial service, legal review comes before technical review.

Finally, self-hosting does not always win. Without sparse attention, the cost of producing a 15-second 2K clip locally may not compare favorably to API pricing. If generation is infrequent and the material is not sensitive, the API is the reasonable choice. The case for self-hosting is usually about the data boundary, not unit price.

Conclusion

MiniMax H3 is an omni-modal model that produces picture and sound in one stream, and its core, H3-Base at 33B, has been released with open weights. Three numbers matter when you evaluate self-hosting: 134 GiB of weights for a single variant, 94,000 tokens for a 15-second 768p clip, and 325,000 tokens when you go to 2K. One condition attaches to all of them: the initial release has no sparse attention, so those sequences must be processed with full attention.

If you take one action, take this one. Rather than deciding whether to adopt H3 now, first confirm whether the material you need video generation for genuinely cannot leave your boundary. If it cannot, a local H3-Base 768p deployment with your own preprocessing pipeline is something you can start today. If it can, you are better off waiting for the sparse attention implementation and the other two modules, then running the numbers again. When that time comes, just rerun the script from this post.

Sources

• Model card: MiniMaxAI/MiniMax-H3 on HuggingFace
• Official announcement: MiniMax H3: An Open Model Breaking the Boundaries Between Tasks and Modalities
• File manifest API: https://huggingface.co/api/models/MiniMaxAI/MiniMax-H3?blobs=true
• Calculation script for this post: scripts/experiments/minimax_h3_serving_budget.py (ThakiCloud internal repository)

What happens when you put written-down skills and baked-in skills into the same forge is this paper’s question.

Why This Matters

This is for people running a skill library behind an agent: you have accumulated hundreds of document skills like SKILL.md files, or you manage per-task LoRA adapters. The conclusion first: the two are not substitutes but complements that cannot cover for each other, and there is a real performance band that appears only when both are fed to one model at once. In the paper’s ablation, the gap between text-only and both-modalities was more decisive than the gap between text-only and weights-only.

Overview

The paper is titled “SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge,” arXiv 2607.27497. Lucio M. Dery and Benedict Aaron Tjandra are joint first authors, with Adhiguna Kuncoro, Arthur Szlam and others making seven contributors. It is filed under cs.CL, and the authors’ own keywords are model merging, kv-caches, continual learning, and prefix tuning.

The situation the paper describes is this. LLM-driven agents learn from past experience along two routes: one synthesizes knowledge in natural language through self-reflection, structured memory, and prompt optimization; the other freezes the behavior of recurring tasks into module weights via PEFT. There is no bridge between them. The text route is flexible but bounded by inference-time context limits. The weight route is efficient at inference, but merging stays at arithmetic like averaging or concatenation and cannot use the semantic relationships between tasks at all.

The paper’s example is intuitive. Suppose an agent trained a prefix cache for English-to-Twi translation in one session, and in another session compiled notes on analyzing English legal documents. Now a new task arrives: analyze a legal document written in Twi. The agent can state in words that this task is a combination of translation ability, Twi language modeling, and legal document analysis. What it cannot do is turn that statement into actual weights. Knowing in words and doing in weights are disconnected.

What the Technique Is

SkillSmith’s idea reduces to one sentence. Treat weights as another modality the LLM already knows how to read.

Concretely, parametric skills are instantiated through prefix-tuning. Choosing prefix over LoRA has a clear reason. A snippet of text becomes a K-V cache the moment you run one forward pass through the base model, which puts text-derived caches and trained caches in the same space. For a goal of bridging two modalities, no representation is more natural.

SkillSmith receives source task bundles. Each bundle is one trained prefix K-V cache plus textual metadata describing that task. On top of that comes combination text describing how the source tasks relate to the target capability. These ingredients are woven into a single sequence using control tokens and passed once through a coprocessor LLM.

flowchart TB
    A["Source task bundle 1<br/>text metadata + prefix K-V"] --> D
    B["Source task bundle 2<br/>text metadata + prefix K-V"] --> D
    C["Combination text<br/>how sources relate to target<br/>+ target task exemplars"] --> D
    D["Input K-V adapter (MLP)<br/>projects weights into the LM latent space"] --> E
    E["Single sequence woven with control tokens<br/>preamble · interleaved bundles · combination · generation block"] --> F
    F["SkillSmith coprocessor<br/>Gemma 3 4B, one forward pass"] --> G
    G["Extract K-V at placeholder positions<br/>strip position info via inverse RoPE"] --> H
    H["Output K-V adapter (MLP)"] --> I["New prefix weights<br/>length 32"]
    I --> J["Attach to frozen base model<br/>solve the target task"]

The sequence has rules. A preamble describing the compositional objective comes first. Then, for each source bundle, the text begins with a <src_start> token and the projected K-V sits between <kv_start> and <kv_end>. After the bundles comes the combination text, and after <gen_start> follows a fixed-length run of placeholder tokens. After the forward pass, only the K-V at those placeholder positions is extracted, stripped of position information via inverse RoPE, and passed through the output adapter to yield the new prefix weights.

Training is end to end. The generated cache is attached to the frozen base model, cross-entropy loss on the target task is computed, and that loss is backpropagated through the base model into SkillSmith. The base model’s weights stay fixed throughout.

The evaluation setup is worth noting. Source task prefix lengths are sampled at random from 32, 64, and 128 to inject diversity, and both the coprocessor and the downstream model are Gemma 3 4B. The number of source tasks is fixed at two. And prefix K-V is trained only on the global attention layers, because a prefix attached to a local attention layer eventually slides out of context.

Installation and Integration

The paper released neither code nor checkpoints, so we could not reproduce SkillSmith itself. Instead we computed what it actually costs to carry one skill under the conditions the paper pinned down: a Gemma 3 4B base, global layers only, prefix length 32. Operationally this number is arguably the one you need before the paper’s Elo figures.

First pull the real Gemma 3 4B settings.

curl -s https://huggingface.co/unsloth/gemma-3-4b-it/raw/main/config.json | jq '.text_config'
# num_hidden_layers: 34, num_key_value_heads: 4, head_dim: 256,
# sliding_window_pattern: 6, torch_dtype: "bfloat16"

Because sliding_window_pattern is 6, one layer in every six is global attention. Of 34 layers, five are global and the remaining twenty-nine are local. The paper pins prefix training to global layers only, so the parameters actually trained cover five layers.

The calculation script lives in the ThakiCloud repository.

.venv/bin/python scripts/experiments/skillsmith_prefix_budget.py

It measures two things together. One is the parameter count and byte size of a single prefix K-V under the settings above. The other is the real size of a local skill corpus. The ThakiCloud workspace currently holds 1,911 SKILL.md files under .claude/skills/. Loading a text skill into context means those tokens also become resident K-V, which lets us compare both approaches on the same unit.

Measured Results

Our own budget first.

On the text side: the median of 1,911 local SKILL.md files is 6,173 characters, which at a conservative 4 characters per token is roughly 1,543 tokens. Keeping that resident in a Gemma 3 4B context puts K-V on all 34 layers, about 205 MiB. Against 640 KiB for the same skill carried parametrically, that is a factor of 328. By sequence position it is 1,543 versus 32, a factor of 48.

Left is the ablation from the paper’s Table 1; right is the residency cost computed by the script above.

Now the paper’s numbers. To check whether SkillSmith really uses the K-V caches, the authors removed inputs one at a time. Elo measured on the Composite-SNI meta-evaluation tasks:

How to read this matters. Text only (1622) beats weights only (1455), meaning text is the richer signal. But both together reach 1714, which is 92 points above text alone. Those 92 points earned by adding weights are this paper’s central claim. As the authors note, the K-V-only setting is effectively equivalent to the existing ATTEMPT approach of combining PEFT modules with a parametric function instead of arithmetic. So this table lays the ceiling of existing weight-merging methods along the bottom and shows the lift from adding text.

The authors also tested the obvious objection that the gain merely comes from having more text. They extracted all source text and combination text fed to SkillSmith, prepended it directly to the target task inputs, and trained an ordinary prefix cache. Auxiliary text did improve the baseline, but it did not catch SkillSmith. Text simply being present and text being synthesized together with weights are different things.

There is a generalization check too. The 15 meta-evaluation tasks were re-scored split by whether neither, one, or both parent tasks appeared in meta-training, and SkillSmith led in all three. Winning by a large margin even where it had never seen the parent tasks is the notable part.

It did not win on every dataset, however. On the real SNI dataset, once downstream fine-tuning was allowed, the win rate among the top three methods converged to 0.5. The authors attribute this to SNI having on the order of 1,000 instances per task and, having been built in 2022, consisting mostly of primitive work like sentiment classification and character concatenation that is trivial for contemporary models. On MMLU-ProX, with only about 250 tasks and higher difficulty, the gap reopened. There, the variant trained on MMLU-ProX alone without bootstrapping from a Composite-SNI checkpoint reached only 1736 zero-shot Elo.

What This Means for ThakiCloud Products

This paper touches both ThakiCloud products.

The Paxis lens comes first. Paxis is ThakiCloud’s Agent-Native Cloud control plane, treating Skills, Tools, Policies, and Audit Logs as first-class resources. At its center, the Skill Harness currently selects text skills via BM25 and runs them in an isolated sandbox. This paper touches exactly that structure’s cost curve, and the factor of 328 measured above is the reason. As long as skills are carried as text, context budget decides how many can be active at once. That is why a router is needed, and when the router is wrong the turn simply fails.

Parametric skills move that constraint to a different axis. With a 32-position prefix, eight active skills is 256 positions and 5 MiB. To be honest, though, this is complementary rather than a replacement. The paper’s own ablation says text-only beats weights-only, so abandoning text skills for weights is not a direction this paper supports. What it supports is placing a parametric counterpart alongside the text skill the router selected. Since Paxis’s self-evolving skill pipeline already creates, revises, and evaluates skills, adding a prefix cache to that pipeline’s outputs is an extension of an existing artifact list rather than a new architecture.

The ai-platform lens is about serving. A prefix cache is a per-request adapter, not a model. Keeping thousands of 640 KiB objects per tenant and attaching one on request is structurally the same as multi-adapter LoRA serving. ThakiCloud’s ai-platform already runs multi-tenant vLLM serving on K8s and Kueue, so sharing one base model while attaching a different skill set per tenant does not depart far from the current design. This matters most in on-premise and sovereign environments. Instead of fine-tuning a separate model per customer and consuming GPUs for each, you share one base model and manage per-customer skills as KiB-scale objects, so GPU occupancy no longer scales with customer count.

The two lenses connect. When skills are cheap to carry (ai-platform), an agent can carry more of them (Paxis), and when it carries more, being wrong about which one to pick costs less.

Limits and Counterarguments

The biggest constraint is reproducibility. Neither code nor checkpoints were released, and SkillSmith requires end-to-end meta-training that backpropagates through the base model. You must first build a per-task prefix library, and the combination text has to be generated with Gemini 2.5. The description of roughly 350K initial synthetic tasks suggests the preparation alone is a substantial investment. It is early to expect the same results in your own domain.

Second, the experimental scale is small. Both the coprocessor and downstream model are a single 4B model, and the meta-evaluation task counts are 15, 10, and 6. Elo is a relative metric that says nothing about absolute performance, and its values shift when the candidate set changes. The number of source tasks is fixed at two, so this paper alone cannot tell us whether the approach extends to the multi-skill compositions real agents face.

Third, the convergence seen on SNI narrows the applicable range. When data is plentiful and tasks are easy, direct training reaches the same place. SkillSmith earns its keep where tasks are hard and instances are scarce. Plenty of in-house domain work fits that description, but there is no reason to route every task through this method.

Finally, the factor of 328 computed above is a difference in storage and residency, not in capability. It absolutely does not mean a 640 KiB prefix does the same job as a 205 MiB text context. The paper’s ablation says precisely the opposite, and the calculation here exists to put both approaches on the same ruler, not to justify a replacement.

Conclusion

For a long time, giving an agent a skill meant choosing between documents and weights. This paper puts a third answer on the table. Make weights readable to an LLM, and the relationship descriptions you wrote down become instructions that produce actual weights. The 92 points in the ablation are evidence that those instructions were not empty talk.

If you are running a skill library today, check one thing. Do your skill documents contain a sentence describing how this skill connects to other skills? What SkillSmith actually consumed was not the task description but the description of relationships between tasks. If that sentence is missing, the most important ingredient will be absent when you later try to attach parametric synthesis. If it is already there, baking one more prefix cache into your skill pipeline is a closer next step than it looks.

Sources

• Paper: SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge (arXiv:2607.27497)
• Authors: Lucio M. Dery, Benedict Aaron Tjandra, Siavash Samiei, Adhiguna Kuncoro, Zohar Yahav, Jiajun Shen, Arthur Szlam (Google DeepMind), submitted 29 July 2026
• Base model config: Gemma 3 4B config.json
• Calculation script for this post: scripts/experiments/skillsmith_prefix_budget.py (ThakiCloud internal repository)

영상과 소리를 따로 만들어 붙이는 대신 하나의 시퀀스에서 함께 뽑는다는 것이 H3 설계의 출발점입니다.

왜 읽어야 하나

사내 GPU 클러스터를 운영하면서 영상 생성 모델을 자체 호스팅할지 API를 쓸지 판단해야 하는 분을 위한 글입니다. 결론을 먼저 말씀드리면, H3-Base는 33B 파라미터 단일 스트림 트랜스포머라 숫자만 보면 LLM 서빙과 비슷해 보이지만 실제 병목은 가중치가 아니라 시퀀스 길이에 있습니다. 15초짜리 2K 클립 하나가 32만 토큰이 넘는 시퀀스를 만들고, 초기 공개본에는 이를 감당할 sparse attention 구현이 빠져 있습니다.

개요

MiniMax H3는 범용 옴니모달 생성 시스템입니다. 텍스트와 이미지, 영상, 오디오가 뒤섞인 멀티모달 컨텍스트를 통합적으로 이해하고, 거기서 스테레오 오디오가 붙은 영상을 생성합니다. 출력 사양은 4초에서 15초 길이, 24 FPS, 32 kHz 스테레오 오디오이고 기본 짧은 변 해상도는 768픽셀입니다. 21대9부터 9대16까지 다양한 화면비를 지원하고 대사는 아랍어와 한국어를 포함해 11개 언어를 안정적으로 지원합니다.

기존 영상 생성 모델과의 차이는 모달리티를 실로 꿰맨 자리가 없다는 점입니다. 텍스트와 시각, 오디오 토큰이 각자의 사일로에 있다가 마지막에 합쳐지는 것이 아니라 하나의 트랜스포머 스트림을 공유합니다. 모델 카드의 표현을 빌리면 어텐션 레이어에도 FFN 레이어에도 모달리티 전용 구조가 없고, 모달리티별 파라미터는 입출력 레이어와 AdaLN 분기에만 있습니다.

라이선스는 MiniMax H3 Community License입니다. Apache나 MIT 같은 표준 오픈소스 라이선스가 아니라 자체 커뮤니티 라이선스이므로, 상업적 배포를 검토하신다면 조항을 먼저 읽어 보셔야 합니다.

이 기술은 무엇인가

H3는 하나의 모델이 아니라 세 개의 모듈로 이루어진 시스템입니다. 이 구분이 자체 호스팅을 계획할 때 가장 먼저 알아야 할 사실입니다. 세 모듈 중 오픈소스로 풀린 것은 가운데 하나뿐이기 때문입니다.

flowchart TB
    A["사용자 입력<br/>텍스트 · 이미지 최대 9장<br/>영상 최대 3클립 · 오디오 최대 3클립"] --> B
    B["H3-Context-IR<br/>지시 파싱 · 교차모달 연결 · 시간 이해<br/>미공개, API로만 제공"] --> C
    C["Context Intermediate Representation<br/>구조화된 중간 표현"] --> D
    D["H3-Base<br/>H3-Encoder + H3-VisualVAE + H3-AudioVAE<br/>+ H3-Omni-Transformer 33B<br/>오픈웨이트 공개"] --> E
    E["768p 결과물<br/>영상 + 32kHz 스테레오"] --> F
    F["H3-Regenerate-2K<br/>원본 컨텍스트와 함께 재생성<br/>미공개, API로만 제공"] --> G["2K 최종 출력"]
    E -.로컬 배포로 여기까지 가능.-> H["자체 호스팅 도달 범위"]

H3-Context-IR은 자유 형식 멀티모달 입력을 전처리하고 조율하는 호스팅 시스템입니다. 여러 단계의 워크플로와 여러 호스팅 모델에 의존하기 때문에 이번 공개에 포함되지 않았습니다. MiniMax는 공식 워크플로를 재현할 수 있는 API와 직접 전처리 시스템을 만들 수 있는 프롬프팅 가이드를 함께 제공합니다. 다만 모델 카드가 명시적으로 밝히듯 H3-Context-IR은 최종 출력 품질에 결정적이므로, 이것을 빼고 H3-Base만 돌리면 공식 데모와 같은 결과를 기대하기 어렵습니다.

H3-Regenerate-2K도 미공개입니다. 흥미로운 점은 이것이 별도의 초해상도 모듈이 아니라는 것입니다. 768p 결과물을 원본 멀티모달 컨텍스트와 함께 H3에 다시 넣어 2K로 재생성합니다. 통상적인 초해상도가 추측으로 채워야 하는 작은 글자나 미세한 디테일을 원본 컨텍스트에서 복원할 수 있다는 것이 이 설계의 장점입니다.

가운데의 H3-Base가 공개된 부분입니다. 구성은 이렇습니다. 텍스트는 H3-Encoder가 인코딩하는데, 이것은 Qwen3-VL-32B의 사전학습 가중치를 그대로 쓰고 50번째 레이어의 은닉 상태를 트랜스포머에 넘깁니다. 시각 입력은 H3-Encoder와 H3-VisualVAE 양쪽을 거치고 오디오는 H3-AudioVAE만 거칩니다. 그리고 H3-Omni-Transformer가 영상과 오디오 잠재를 동시에 예측합니다.

VAE 사양이 나중에 계산에 쓰이니 적어 두겠습니다. H3-VisualVAE는 시간적으로 인과적인 영상 오토인코더로 공간 압축 16배, 시간 압축 4배, 잠재 채널 24개입니다. 여기에 시간과 높이, 너비 축으로 1대2대2 패치화가 더해지므로 트랜스포머에 들어가는 시각 토큰의 유효 공간 압축률은 32배가 됩니다. 시간 압축률은 4배 그대로입니다. H3-AudioVAE는 좌우 채널에 같은 인코더와 디코더를 쓰되 각 채널을 독립 처리하며, 32 kHz 오디오를 채널당 40 Hz 잠재 토큰 열로 압축합니다.

설치 및 통합

전체 무게를 먼저 봅니다. HuggingFace API로 파일 매니페스트를 받아 safetensors만 합산했습니다.

curl -s "https://huggingface.co/api/models/MiniMaxAI/MiniMax-H3?blobs=true" \
  | jq '[.siblings[] | select(.rfilename|endswith(".safetensors"))
         | {f:.rfilename, b:(.lfs.size // .size)}]'

계산 스크립트는 다키클라우드 저장소에 두었습니다. 모듈별 바이트 합산과 시퀀스 길이 유도를 함께 수행합니다.

.venv/bin/python scripts/experiments/minimax_h3_serving_budget.py

Diffusers로 불러오는 최소 경로는 모델 카드에 나와 있습니다.

pip install -U diffusers transformers accelerate

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "MiniMaxAI/MiniMax-H3", dtype=torch.bfloat16, device_map="cuda"
)

정직하게 밝혀 둡니다. 이번 작업에서 H3-Base를 실제로 로드해 추론하지는 못했습니다. 아래에 필요한 자원을 계산해 두었지만 그만한 단일 노드를 이 작업에 붙이지 않았기 때문입니다. 따라서 이 글에는 생성 품질이나 실측 지연 시간에 관한 수치가 없습니다. 있는 것은 공개된 파일 크기와 공개된 압축 계수에서 결정론적으로 유도한 자원 수치뿐입니다.

실제 실험 결과

리포지토리 전체의 safetensors 합계는 464 GiB입니다. 다만 이 숫자를 그대로 필요 용량으로 읽으면 안 됩니다. FL2VA와 Ref2VA, 그리고 루트 레벨에 같은 가중치가 여러 레이아웃으로 배치돼 있기 때문입니다. 실제로 한 벌을 배포할 때 필요한 크기는 이렇습니다.

바이트에서 환산한 33.14B가 모델 카드가 밝힌 33B와 일치합니다. 계산 경로가 맞다는 확인입니다.

여기서 모델 카드의 한 문장이 중요해집니다. H3-Omni-Transformer의 33B 중 약 13B가 AdaLN 관련 분기에 있는데, AdaLN 변조 출력은 미리 계산해 캐시할 수 있으므로 추론 전용 배포에서는 이 파라미터를 로드할 필요가 없습니다. 전체 가중치를 공개한 것은 파인튜닝을 포함한 후속 개발을 지원하기 위해서입니다. 그러니 추론만 하실 계획이라면 트랜스포머 쪽은 약 20B, bf16 기준 37.5 GiB로 줄어듭니다.

왼쪽은 파일 매니페스트에서 합산한 모듈별 가중치이고 오른쪽은 VAE 압축 계수로 유도한 시퀀스 길이입니다.

그런데 진짜 문제는 가중치가 아닙니다. 시퀀스입니다. 위 압축 계수로 계산해 보면 이렇습니다.

읽는 법은 이렇습니다. 15초 클립은 원본 360프레임이지만 시간 압축 4배를 거쳐 잠재 프레임 90개가 됩니다. 768p 16대9를 1376 곱하기 768로 잡으면 유효 공간 압축 32배 후 잠재 프레임당 43 곱하기 24, 즉 1,032개 토큰입니다. 90개 잠재 프레임을 곱하면 92,880개이고 여기에 오디오 1,200개가 더해집니다. 2K로 가면 잠재 프레임당 토큰이 3,600개로 뛰어 합계 32만 5천 개가 됩니다.

토큰 수는 3.5배 늘었는데 풀 어텐션 연산량은 제곱으로 늘어 약 12배가 됩니다. 그리고 모델 카드는 이 지점에서 중요한 사실을 밝힙니다. 긴 멀티모달 시퀀스의 연산 비용을 줄이려고 학습 마지막 단계에 네이티브 sparse attention을 도입했지만, 초기 오픈소스 공개본은 풀 어텐션 추론만 제공하며 sparse attention 구현은 추후 별도로 공개한다는 것입니다.

이것이 자체 호스팅 계획에서 가장 중요한 문장입니다. 공식 API가 내는 결과와 지금 내려받은 가중치로 로컬에서 낼 수 있는 결과 사이에는 품질 차이만 있는 것이 아니라 연산 비용 차이도 있습니다. 9만 4천 토큰짜리 시퀀스를 풀 어텐션으로 처리하는 것은 sparse 구현이 있을 때와는 완전히 다른 예산입니다.

다키클라우드 제품 적용 시사점

다키클라우드의 ai-platform은 K8s와 Kueue 위에서 GPU 워크로드를 스케줄링하고 vLLM 기반 서빙을 멀티테넌트로 운영합니다. H3 같은 모델은 이 구조에 몇 가지 구체적인 요구를 만듭니다.

먼저 배치 단위가 달라집니다. 단일 변형 134 GiB는 H200 한 장의 141 GB에 가중치만으로도 거의 찹니다. 9만 4천 토큰짜리 시퀀스의 활성값과 어텐션 작업 공간까지 생각하면 한 장으로는 부족하고, 추론 전용으로 AdaLN을 덜어내도 여유가 크지 않습니다. 현실적인 구성은 H200 두 장 이상이며, FL2VA와 Ref2VA 두 변형을 모두 서비스하면 그만큼 더 듭니다. Kueue 워크로드로 표현할 때 이 모델은 LLM 서빙 파드와 같은 큐에 놓기 어렵습니다. 요청당 점유 시간이 길고 메모리 프로필이 다르기 때문에 별도 리소스 플레이버로 분리하는 편이 낫습니다.

둘째로 큐 설계가 달라집니다. LLM 서빙은 토큰 단위 스트리밍이라 요청이 짧게 여러 번 지나가지만, 영상 생성은 요청 하나가 GPU를 길게 붙듭니다. 여기에 2K 워크플로는 768p 생성 이후 재생성 단계가 한 번 더 붙습니다. 사용자에게 보이는 응답 시간이 두 단계의 합이 되므로, 대기열 정책은 처리량이 아니라 완료 시간 기준으로 설계해야 합니다.

셋째로 온프렘 수요와 잘 맞습니다. 영상 생성은 소재 자체가 민감한 경우가 많습니다. 촬영 원본, 제품 이미지, 사내 인물이 담긴 영상은 외부 API로 내보내기 어려운 자산입니다. 오픈웨이트 모델을 고객사 경계 안에서 돌릴 수 있다는 점은 그 자체로 요구사항을 만족시킵니다. 다만 H3-Context-IR과 H3-Regenerate-2K가 미공개인 지금 상태에서는 완전한 온프렘 구성이 되지 않습니다. 프롬프팅 가이드를 따라 자체 전처리 시스템을 만들어야 하고, 2K는 API를 타거나 768p로 만족해야 합니다. 이 격차를 고객에게 정확히 설명하는 것이 지금 단계에서 할 수 있는 가장 정직한 일입니다.

Paxis 관점에서 덧붙일 것이 하나 있습니다. H3-Context-IR이 하는 일, 그러니까 자유 형식 멀티모달 입력을 파싱하고 교차모달 관계를 해석해 구조화된 중간 표현으로 직렬화하는 일은 사실 에이전트 오케스트레이션 문제입니다. 이 부분이 미공개라는 것은 반대로 말하면 그 자리를 직접 채울 수 있다는 뜻이기도 합니다. Paxis의 DAG 멀티에이전트 구성과 정책 게이트를 쓰면 입력 정제 파이프라인을 감사 가능한 형태로 만들 수 있고, 어떤 프롬프트 보강이 어떤 출력으로 이어졌는지 추적할 수 있습니다. 생성 결과에 책임을 물어야 하는 조직이라면 이 추적성이 화질보다 먼저 필요한 조건입니다.

한계 및 반론

이 글의 수치에는 분명한 경계가 있습니다. 실제로 모델을 로드해 영상을 뽑아 보지 않았으므로 생성 품질, 실제 지연 시간, 실제 최대 메모리 사용량은 여기 없습니다. 가중치 크기는 파일 매니페스트에서 정확히 나오지만 활성값과 어텐션 작업 공간은 구현에 따라 크게 달라집니다. 따라서 위의 134 GiB는 필요 VRAM의 하한이지 실제 요구량이 아닙니다.

시퀀스 길이 계산도 가정을 하나 깔고 있습니다. 768p 16대9의 실제 픽셀 크기를 1376 곱하기 768로 잡았는데, 모델 카드는 짧은 변이 768이라고만 밝히고 있어 구현에 따라 긴 변이 조금 다를 수 있습니다. 다만 유효 공간 압축 32배와 시간 압축 4배는 공개된 값이므로 자릿수는 흔들리지 않습니다.

라이선스도 짚어야 합니다. MiniMax H3 Community License는 표준 오픈소스 라이선스가 아니고 사용 제한 조항을 포함합니다. 모델 카드는 사용자 제출 텍스트와 이미지, 영상 그리고 보강된 프롬프트가 자동 검열 대상이며 위법하거나 음란하거나 제3자 권리를 침해하는 것으로 의심되는 콘텐츠가 차단될 수 있다고 밝힙니다. 이 안전 가드레일이 라이선스상의 의무를 대신하지 않는다는 점도 명시돼 있습니다. 상업 서비스를 계획하신다면 법무 검토가 기술 검토보다 먼저입니다.

마지막으로 자체 호스팅이 항상 이기는 것은 아닙니다. sparse attention이 빠진 지금 상태에서 15초 2K 클립을 로컬에서 뽑는 비용은 API 호출 단가와 비교했을 때 유리하지 않을 수 있습니다. 생성 빈도가 낮고 소재가 민감하지 않다면 API가 합리적인 선택입니다. 자체 호스팅의 근거는 대개 단가가 아니라 데이터 경계입니다.

정리

MiniMax H3는 영상과 오디오를 한 스트림에서 함께 만드는 옴니모달 모델이고, 그 중심인 H3-Base 33B가 오픈웨이트로 공개됐습니다. 자체 호스팅을 검토하실 때 확인할 숫자는 세 개입니다. 단일 변형 가중치 134 GiB, 15초 768p 클립의 9만 4천 토큰, 그리고 2K로 갈 때의 32만 5천 토큰입니다. 여기에 조건이 하나 붙습니다. 초기 공개본에는 sparse attention이 없어 이 시퀀스를 풀 어텐션으로 처리해야 합니다.

다음 행동을 하나만 고르신다면 이것입니다. 지금 H3를 도입할지 결정하기보다, 사내에서 영상 생성이 필요한 소재가 실제로 외부로 내보낼 수 없는 것인지부터 확인해 보시기 바랍니다. 그렇다면 H3-Base 768p 로컬 배포와 자체 전처리 파이프라인이 지금 바로 시작할 수 있는 구성입니다. 그렇지 않다면 sparse attention 구현과 나머지 두 모듈이 공개되기를 기다렸다가 다시 계산하는 편이 낫습니다. 그때는 이 글의 스크립트를 그대로 다시 돌리시면 됩니다.

출처

• 모델 카드: MiniMaxAI/MiniMax-H3 on HuggingFace
• 공식 소개: MiniMax H3: An Open Model Breaking the Boundaries Between Tasks and Modalities
• 파일 매니페스트 API: https://huggingface.co/api/models/MiniMaxAI/MiniMax-H3?blobs=true
• 이 글의 계산 스크립트: scripts/experiments/minimax_h3_serving_budget.py (다키클라우드 내부 저장소)

글로 적어둔 스킬과 가중치로 구워둔 스킬을 같은 화로에 넣으면 어떻게 되는가가 이 논문의 질문입니다.

왜 읽어야 하나

에이전트에 스킬 라이브러리를 붙여 운영하시는 분, 그러니까 SKILL.md 같은 문서 스킬을 수백 개 쌓아두었거나 태스크별 LoRA 어댑터를 관리하고 계신 분을 위한 글입니다. 결론을 먼저 말씀드리면, 두 방식은 대체재가 아니라 서로를 못 채우는 보완재이며, 한 모델에 둘을 동시에 먹였을 때만 나오는 성능 구간이 실제로 존재합니다. 논문의 절단 실험에서 텍스트만 준 경우와 가중치만 준 경우 사이의 격차보다, 텍스트만 준 경우와 둘 다 준 경우 사이의 격차가 더 결정적이었습니다.

개요

논문 제목은 「SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge」이고 arXiv 번호는 2607.27497입니다. Lucio M. Dery와 Benedict Aaron Tjandra가 공동 1저자이고 Adhiguna Kuncoro, Arthur Szlam 등 일곱 명이 참여했습니다. 분류는 cs.CL이고 저자들이 스스로 붙인 키워드는 model merging, kv-caches, continual learning, prefix tuning 네 개입니다.

논문이 지적하는 현재 상황은 이렇습니다. LLM 기반 에이전트가 과거 경험에서 배우는 경로는 크게 두 가지인데, 하나는 자기반성이나 구조화된 메모리, 프롬프트 최적화처럼 자연어로 지식을 합성하는 경로이고 다른 하나는 PEFT로 반복 작업의 행동을 모듈 가중치에 굳혀 두는 경로입니다. 그런데 이 둘을 잇는 다리가 없습니다. 텍스트 경로는 유연하지만 추론 시점의 컨텍스트 한계에 갇히고, 가중치 경로는 추론 효율은 좋지만 병합 연산이 평균이나 연결 같은 산술에 머물러 태스크 사이의 의미 관계를 전혀 못 씁니다.

논문이 드는 예가 직관적입니다. 어떤 에이전트가 한 세션에서 영어를 트위어로 번역하는 prefix 캐시를 학습했고, 다른 세션에서는 영문 법률 문서 분석에 관한 노트를 정리해 두었다고 합시다. 이제 트위어로 쓰인 법률 문서를 분석하라는 새 작업이 들어옵니다. 에이전트는 이 작업이 번역 능력과 트위어 언어 모델링, 법률 문서 분석의 조합이라는 것을 말로는 설명할 수 있습니다. 그런데 그 설명을 가지고 실제 가중치를 만들어낼 방법이 없습니다. 말로 아는 것과 가중치로 하는 것 사이가 끊겨 있는 것입니다.

이 기술은 무엇인가

SkillSmith의 발상은 한 문장으로 요약됩니다. 가중치를 LLM이 원래 읽을 줄 아는 또 하나의 모달리티로 취급하자는 것입니다.

구체적으로 파라메트릭 스킬을 prefix-tuning으로 구현합니다. LoRA가 아니라 prefix 방식을 고른 데는 분명한 이유가 있습니다. 텍스트 조각은 베이스 모델에 한 번 순전파를 태우면 그대로 K-V 캐시가 되므로, 텍스트에서 유래한 캐시와 학습으로 만든 캐시가 같은 공간에 놓입니다. 두 모달리티를 잇겠다는 목표에 이보다 자연스러운 표현은 없습니다.

SkillSmith는 소스 태스크 묶음을 받습니다. 각 묶음은 학습된 prefix K-V 캐시 하나와 그 태스크를 설명하는 텍스트 메타데이터 하나로 이루어집니다. 여기에 소스 태스크들이 목표 능력과 어떻게 연결되는지를 서술한 조합 텍스트가 더해집니다. 이 재료들을 제어 토큰으로 엮어 하나의 시퀀스로 만들고 코프로세서 LLM에 한 번 통과시킵니다.

flowchart TB
    A["소스 태스크 묶음 1<br/>텍스트 메타데이터 + prefix K-V"] --> D
    B["소스 태스크 묶음 2<br/>텍스트 메타데이터 + prefix K-V"] --> D
    C["조합 텍스트<br/>소스와 목표의 관계 서술<br/>+ 목표 태스크 예시"] --> D
    D["입력 K-V 어댑터 (MLP)<br/>가중치를 언어 모델 잠재 공간으로 사영"] --> E
    E["제어 토큰으로 엮은 단일 시퀀스<br/>서문 · 묶음 교차 배치 · 조합문 · 생성 블록"] --> F
    F["SkillSmith 코프로세서<br/>Gemma 3 4B, 순전파 1회"] --> G
    G["생성 블록 자리표시 토큰의 K-V 추출<br/>역 RoPE로 위치 정보 제거"] --> H
    H["출력 K-V 어댑터 (MLP)"] --> I["새 prefix 가중치<br/>길이 32"]
    I --> J["동결된 베이스 모델에 부착<br/>목표 태스크 수행"]

시퀀스 구조에는 규칙이 있습니다. 맨 앞에 조합 목표를 설명하는 서문 텍스트가 오고, 그다음 소스 묶음마다 텍스트가 <src_start> 토큰으로 시작해 사영된 K-V가 <kv_start>와 <kv_end> 사이에 놓입니다. 묶음들이 끝나면 조합 텍스트가 붙고, <gen_start> 이후에 고정 길이의 자리표시 토큰들이 이어집니다. 순전파 뒤 이 자리표시 토큰 위치의 K-V만 뽑아내 역 RoPE로 위치 정보를 벗기고 출력 어댑터를 통과시키면 새 prefix 가중치가 나옵니다.

학습은 종단간입니다. 생성된 캐시를 동결된 베이스 모델에 붙여 목표 태스크의 교차 엔트로피 손실을 계산하고, 그 손실을 베이스 모델을 거쳐 SkillSmith 쪽으로 역전파합니다. 베이스 모델 가중치는 끝까지 고정입니다.

평가 설정도 눈여겨볼 만합니다. 소스 태스크의 prefix 길이는 32, 64, 128 중에서 무작위로 뽑아 다양성을 넣었고, 코프로세서와 다운스트림 모델 모두 Gemma 3 4B를 씁니다. 소스 태스크 개수는 두 개로 고정했습니다. 그리고 prefix K-V는 전역 어텐션 레이어에만 학습합니다. 지역 어텐션 레이어에 붙인 prefix는 슬라이딩 윈도가 지나가면 결국 컨텍스트 밖으로 밀려나기 때문입니다.

설치 및 통합

논문은 코드나 체크포인트를 공개하지 않았습니다. 그래서 SkillSmith 자체를 재현하지는 못했습니다. 대신 논문이 못 박아 둔 설정, 그러니까 Gemma 3 4B 베이스에 전역 레이어만, prefix 길이 32라는 조건에서 스킬 하나를 들고 다니는 비용이 실제로 얼마인지를 계산했습니다. 이 값은 운영 관점에서 논문의 Elo 수치보다 오히려 먼저 알아야 하는 숫자입니다.

Gemma 3 4B의 실제 설정값을 먼저 받아옵니다.

curl -s https://huggingface.co/unsloth/gemma-3-4b-it/raw/main/config.json | jq '.text_config'
# num_hidden_layers: 34, num_key_value_heads: 4, head_dim: 256,
# sliding_window_pattern: 6, torch_dtype: "bfloat16"

sliding_window_pattern이 6이므로 여섯 개 레이어마다 하나가 전역 어텐션입니다. 34개 레이어 중 전역은 다섯 개, 나머지 스물아홉 개는 지역입니다. 논문이 전역 레이어에만 prefix를 학습한다고 못 박았으니 실제로 학습되는 파라미터는 다섯 개 레이어 몫입니다.

계산 스크립트는 다키클라우드 저장소에 두었습니다.

.venv/bin/python scripts/experiments/skillsmith_prefix_budget.py

이 스크립트는 두 가지를 함께 잽니다. 하나는 위 설정에서 prefix K-V 하나의 파라미터 수와 바이트이고, 다른 하나는 로컬 스킬 코퍼스의 실제 크기입니다. 다키클라우드 워크스페이스에는 현재 .claude/skills/ 아래에 SKILL.md가 1,911개 있습니다. 텍스트 스킬을 컨텍스트에 올리면 그 토큰들도 결국 K-V로 상주하므로, 두 방식을 같은 단위로 놓고 비교할 수 있습니다.

실제 실험 결과

먼저 우리가 계산한 예산입니다.

텍스트 쪽은 이렇습니다. 로컬 SKILL.md 1,911개의 중앙값은 6,173자이고 4자를 1토큰으로 보수적으로 환산하면 약 1,543토큰입니다. 이 분량을 Gemma 3 4B 컨텍스트에 상주시키면 전체 34개 레이어에 K-V가 잡히므로 약 205 MiB가 됩니다. 같은 스킬 하나를 파라메트릭으로 들고 있을 때의 640 KiB와 비교하면 328배 차이입니다. 시퀀스 위치 기준으로 봐도 1,543개 위치와 32개 위치이니 48배입니다.

왼쪽은 논문 Table 1의 절단 실험이고 오른쪽은 위 스크립트가 계산한 상주 비용입니다.

논문 쪽 수치도 정리해 두겠습니다. 저자들은 SkillSmith가 정말로 K-V 캐시를 쓰는지 확인하려고 입력을 하나씩 지워 봤습니다. Composite-SNI 메타평가 태스크에서 측정한 Elo는 다음과 같습니다.

읽는 법이 중요합니다. 텍스트만 준 경우(1622)가 가중치만 준 경우(1455)보다 낫습니다. 텍스트가 더 풍부한 신호라는 뜻입니다. 그런데 둘 다 준 경우는 1714로, 텍스트만 준 경우보다 92점 더 높습니다. 가중치를 더해서 얻은 이 92점이 이 논문의 핵심 주장입니다. 저자들이 지적하듯 K-V 캐시만 쓰는 설정은 PEFT 모듈을 산술 대신 파라메트릭 함수로 합치는 기존 ATTEMPT 방식과 사실상 같습니다. 그러니 이 표는 기존 가중치 병합 계열이 어디까지 갈 수 있는지를 아래쪽에 깔아두고, 텍스트를 더했을 때의 상승 폭을 보여주는 셈입니다.

성능 상승이 단지 텍스트를 더 줘서 생긴 것 아니냐는 반론도 저자들이 직접 검증했습니다. SkillSmith에 들어간 소스 텍스트와 조합 텍스트를 전부 뽑아 목표 태스크 입력 앞에 그대로 붙이고 일반 prefix 캐시를 학습시켜 봤는데, 보조 텍스트가 기준선을 개선하기는 했지만 SkillSmith를 따라잡지는 못했습니다. 텍스트가 그냥 거기 있는 것과 가중치와 함께 합성되는 것은 다르다는 뜻입니다.

일반화 검증도 있습니다. 15개 메타평가 태스크를 부모 태스크가 메타학습에 전혀 등장하지 않은 경우, 하나만 등장한 경우, 둘 다 등장한 경우로 나눠 다시 재보니 세 구간 모두에서 SkillSmith가 앞섰습니다. 부모 태스크를 한 번도 못 본 구간에서도 큰 격차로 이겼다는 점이 특히 눈에 띕니다.

다만 모든 데이터셋에서 이긴 것은 아닙니다. 실제 SNI 데이터셋에서는 다운스트림 미세조정을 허용하자 상위 세 방법의 승률이 0.5로 수렴했습니다. 저자들은 SNI 태스크가 태스크당 인스턴스가 1,000개 수준으로 많고, 2022년에 만들어져 감성 분류나 문자 연결 같은 원시적 작업 위주라 요즘 모델에게는 너무 쉽기 때문이라고 봅니다. 반대로 태스크가 250개뿐이고 난도가 높은 MMLU-ProX에서는 격차가 다시 벌어졌습니다. 여기서 Composite-SNI 체크포인트로 부트스트랩하지 않고 MMLU-ProX만으로 학습한 변형은 제로샷 Elo 1736에 그쳤습니다.

다키클라우드 제품 적용 시사점

이 논문은 다키클라우드의 두 제품 모두에 걸칩니다.

Paxis 관점이 먼저입니다. Paxis는 다키클라우드의 Agent-Native Cloud 제어 평면으로, Skills와 Tools, Policies, Audit Logs를 일급 리소스로 다룹니다. 그 중심에 있는 Skill Harness는 현재 텍스트 스킬을 BM25로 골라 격리 샌드박스에서 실행하는 구조입니다. 이 논문이 정확히 그 구조의 비용 곡선을 건드립니다. 위에서 잰 328배라는 숫자가 그것입니다. 스킬을 텍스트로 들고 있는 한, 동시에 켤 수 있는 스킬 수는 컨텍스트 예산이 결정합니다. 그래서 라우터가 필요하고, 라우터가 틀리면 그 턴은 그냥 실패합니다.

파라메트릭 스킬은 이 제약을 다른 축으로 옮깁니다. 32개 위치짜리 prefix라면 여덟 개를 동시에 얹어도 256 위치, 5 MiB입니다. 다만 정직하게 말씀드리면 이것은 아직 교체가 아니라 보완입니다. 논문 자신의 절단 실험이 텍스트만 쓴 쪽이 가중치만 쓴 쪽보다 낫다고 말하고 있으니, 텍스트 스킬을 버리고 가중치로 옮기는 것은 논문이 지지하지 않는 방향입니다. 이 논문이 지지하는 것은 라우터가 고른 텍스트 스킬 옆에 그 스킬로 학습된 파라메트릭 대응물을 함께 놓는 구조입니다. Paxis의 자가진화 스킬 파이프라인이 이미 스킬을 만들고 고치고 평가하고 있으므로, 그 파이프라인의 산출물에 prefix 캐시를 하나 더 붙이는 것은 새 아키텍처가 아니라 기존 산출물 목록의 확장입니다.

ai-platform 관점은 서빙 쪽입니다. prefix 캐시는 요청마다 바뀌는 어댑터이지 모델이 아닙니다. 640 KiB짜리 객체를 테넌트별로 수천 개 보관하다가 요청이 오면 붙이는 일은 LoRA 멀티 어댑터 서빙과 형태가 같습니다. 다키클라우드의 ai-platform은 K8s와 Kueue 위에서 vLLM 서빙을 멀티테넌트로 운영하고 있으므로, 테넌트마다 다른 스킬 세트를 붙여 같은 베이스 모델을 공유하는 구성은 지금 구조에서 크게 벗어나지 않습니다. 온프렘이나 소버린 환경에서는 이 점이 특히 중요합니다. 고객사마다 별도 모델을 파인튜닝해 각각 GPU를 잡아먹는 대신, 베이스 모델 하나를 공유하고 고객사별 스킬만 KiB 단위 객체로 관리하면 GPU 점유가 고객사 수에 비례해 늘지 않습니다.

두 렌즈는 이어집니다. 스킬을 싸게 들고 다닐 수 있으면(ai-platform) 에이전트가 들고 다닐 수 있는 스킬 수가 늘고(Paxis), 스킬 수가 늘면 라우터가 틀렸을 때의 손해가 줄어듭니다.

한계 및 반론

가장 큰 제약은 재현 가능성입니다. 코드도 체크포인트도 공개되지 않았고, SkillSmith는 베이스 모델을 거쳐 역전파하는 종단간 메타학습을 요구합니다. 태스크별 prefix 라이브러리를 먼저 만들어야 하고, 조합 텍스트도 Gemini 2.5로 생성해야 합니다. 초기 합성 태스크가 35만 개 규모라는 서술을 보면 이 준비 과정 자체가 상당한 투자입니다. 그러니 이 결과를 자체 도메인에서 그대로 얻을 수 있으리라 기대하기는 이릅니다.

둘째로 실험 규모가 작습니다. 코프로세서와 다운스트림 모두 4B 모델 하나이고, 메타평가 태스크는 각각 15개, 10개, 6개입니다. Elo는 상대 비교 지표이므로 절대 성능을 말해 주지 않으며, 후보 집합이 바뀌면 값도 바뀝니다. 소스 태스크 개수도 두 개로 고정돼 있어 실제 에이전트가 마주하는 다중 스킬 조합으로 바로 확장되는지는 이 논문만으로 알 수 없습니다.

셋째로 SNI 결과가 보여준 수렴 현상은 적용 범위를 좁힙니다. 데이터가 넉넉하고 태스크가 쉬우면 그냥 직접 학습해도 같은 곳에 도달합니다. SkillSmith가 값어치를 하는 구간은 태스크가 어렵고 인스턴스가 적은 데이터 희소 환경입니다. 사내 도메인 작업 상당수가 여기 해당하기는 하지만, 모든 작업을 이 방식으로 처리할 이유는 없습니다.

마지막으로 위에서 계산한 328배는 저장과 상주 비용의 차이이지 성능의 차이가 아닙니다. 640 KiB짜리 prefix가 205 MiB짜리 텍스트 컨텍스트와 같은 일을 한다는 뜻이 절대 아닙니다. 논문의 절단 실험이 말하는 바가 정확히 그 반대이고, 이 글의 계산은 두 방식을 같은 자로 재보기 위한 것이지 교체를 정당화하기 위한 것이 아닙니다.

정리

에이전트에 스킬을 붙이는 문제에서 우리는 오랫동안 문서냐 가중치냐를 골라 왔습니다. 이 논문은 그 질문에 세 번째 답을 놓습니다. 가중치를 LLM이 읽을 수 있는 입력으로 만들어 두면, 문서로 적어둔 관계 서술이 실제 가중치를 만드는 지시문이 됩니다. 절단 실험의 92점은 그 지시문이 헛말이 아니었다는 증거입니다.

지금 스킬 라이브러리를 운영하고 계신다면, 다음 한 가지를 확인해 보시기를 권합니다. 여러분의 스킬 문서에 이 스킬이 다른 스킬과 어떻게 이어지는지를 서술한 문장이 있습니까. SkillSmith가 실제로 소비한 것은 태스크 설명이 아니라 태스크 사이의 관계 서술이었습니다. 그 문장이 없다면 나중에 파라메트릭 합성을 붙이려 할 때 가장 중요한 재료가 비어 있는 셈입니다. 반대로 그 문장이 이미 있다면, 스킬 파이프라인에 prefix 캐시를 하나 더 굽는 일은 생각보다 가까운 다음 단계입니다.

출처

• 논문: SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge (arXiv:2607.27497)
• 저자: Lucio M. Dery, Benedict Aaron Tjandra, Siavash Samiei, Adhiguna Kuncoro, Zohar Yahav, Jiajun Shen, Arthur Szlam (Google DeepMind), 2026년 7월 29일 제출
• 베이스 모델 설정: Gemma 3 4B config.json
• 이 글의 계산 스크립트: scripts/experiments/skillsmith_prefix_budget.py (다키클라우드 내부 저장소)

Why read this

This is for platform engineers who run skills or plugins on coding agents, and for anyone who has to choose a skill format for an in-house agent. Two conclusions up front. First, the skill file format has already converged past vendor boundaries. Second, a skill file that actually works is a body of code and numeric constraints, not prose. The Google Antigravity skill we measured is more than half code across its 314 lines.

The occasion was ordinary. A content creator published a tutorial on building animated websites with Gemini 3.6 Flash and Antigravity, and a repository encapsulating that workflow as a skill appeared on GitHub. The file structure turned out to be far more interesting than the tutorial, because it is effectively the same specification we use every day.

Overview

The subject is WilkoMarketing/antigravity-video-websites-skill, an Antigravity skill that turns a video file into a scroll-driven animated website. The repository describes itself as a “Google Antigravity skill to turn videos into premium animated scroll-driven websites.”

We do three things here. We take the skill file apart, we measure our own 1911 production skills with exactly the same yardstick, and we work out what the token economics of Gemini 3.6 Flash mean for agent unit cost. We did not install the Antigravity IDE and run it, so this is a structural analysis of a published artifact rather than an execution benchmark.

What an Antigravity skill is made of

The install instructions give the structure away. The global skills directory is ~/.gemini/antigravity/skills/ on macOS and Linux, with one folder per skill.

.gemini/antigravity/skills/creating-video-websites/
├── SKILL.md
└── scripts/
    └── extract_frames.py

SKILL.md opens like this.

---
name: creating-video-websites
description: Turn a video into a premium scroll-driven animated website with GSAP, canvas frame rendering, and layered animation choreography. Use when the user wants to convert a video into an animated web experience.
---

Two keys, name and description, where the description states the capability in one sentence and then attaches a trigger condition with “Use when.” That matches, sentence for sentence, the skill description contract we hold as an internal rule. Different vendor, different IDE, same specification.

The body has four top-level sections: When to use this skill, Input, Premium Checklist (Non-Negotiable), and Workflow. The checklist is the part worth studying. It converts design quality, which looks like a matter of taste, into sixteen numeric constraints. Hero typography at 12rem or more, marquee text at 10vw or more, total scroll height of 800vh or more for six sections, stats overlay opacity between 0.88 and 0.92, frame advance speed between 1.8 and 2.2, canvas image scale with a sweet spot of 0.82 to 0.90. Another clause requires at least four animation types and forbids repeating the same entrance effect consecutively.

The workflow has seven steps, and step zero stands out. Step 0: The Interview (MANDATORY) forces the agent to ask the user six questions covering brand name, logo, accent color, background color, overall vibe, and content source before extracting a single frame or writing a line of code. It is a structural guard against an agent inventing requirements and proceeding.

The remaining steps specify the entire skeleton of the output. Step one slices the video into 150 to 300 WebP frames using the bundled script, optionally running rembg for background removal via the --remove-bg flag. Step two scaffolds into index.html, css/style.css, js/app.js, and frames/, with no bundler, just vanilla HTML, CSS, and JS plus CDN libraries. From step three onward it pins the order of loader, navigation, fixed canvas, and marquee text, how Lenis drives smooth scroll and connects to the GSAP ticker, and how the canvas renderer samples its background color from frame edge pixels every twenty frames. Dependencies are explicit too: opencv-python and numpy, plus rembg[cpu] if background removal is used.

flowchart TB
    A["User request<br/>build me a site from this video"] --> B["Skill selected via<br/>the Use when clause in description"]
    B --> C["Step 0 mandatory interview<br/>brand · colors · vibe · content source"]
    C --> D["Run bundled script<br/>scripts/extract_frames.py"]
    D --> E["Apply Non-Negotiable checklist<br/>16 numeric constraints"]
    E --> F["Generate from code templates<br/>index.html · style.css · app.js"]
    F --> G["Scroll-driven animated site"]

The execution path. Model freedom is squeezed between the checklist and the code templates.

How we measured it

Instead of impressions we counted. A script downloads the file and counts lines, code fence lines, frontmatter keys, and checklist items.

for l in lines:
    if l.strip().startswith("```"):
        in_fence = not in_fence
        code += 1
        continue
    if in_fence:
        code += 1

The same function then ran across every .claude/skills/*/SKILL.md in our repository to build a comparison set. One trap surfaced. Extracting the description with ^description:\s*(.*)$ captures only >- for the YAML folded style most of our skills use, dropping the body entirely. The first run reported only 71 of 1911 skills carrying a “Use when” trigger, which was a parser bug rather than a property of the corpus. After teaching it to join the indented continuation lines of a folded block, the number landed where it should.

inline = line.partition(":")[2].strip()
if inline and inline not in (">", ">-", "|", "|-", ">+", "|+"):
    return inline
# folded block: join the indented continuation lines

The scripts are at scripts/blog/_skillmd_anatomy_20260803.py and _skillmd_corpus_20260803.py, with raw output under outputs/blog-impl/antigravity-skill-format-gemini-flash/.

Results

For the Antigravity skill: SKILL.md is 13,735 bytes across 314 lines, of which 171 lines sit inside code fences, or 54.5 percent. The frontmatter carries exactly two keys, name and description, the description is 205 characters and contains “Use when.” There are four top-level sections, seven workflow steps, sixteen checklist items, and the bundled scripts/extract_frames.py is 84 lines.

Our own corpus, across 1911 skills, has a median of 154 lines, a mean of 189.3, and a maximum of 2063. Code share runs to a median of 18.5 percent and a mean of 19.5 percent. 1379 skills (72.2 percent) carry a “Use when” trigger in their description and 1396 (73.1 percent) keep frontmatter to just name and description. Only 154 (8.1 percent) ship a bundled scripts/ directory.

Code fence line share of SKILL.md under identical counting rules. The Antigravity skill’s 54.5 percent sits well above our corpus median of 18.5 percent but below our toolkit-class skills, which prescribe execution steps most tightly.

The comparison sharpens the picture. The Antigravity skill is twice as long as our median and close to three times as code-dense. Yet within our own corpus the skills that prescribe execution most tightly run denser still: pillow-toolkit at 679 lines and 87.3 percent, exiftool-toolkit at 519 lines and 85.9 percent, vips-toolkit at 483 lines and 77.0 percent. The median sits at 18.5 percent because the corpus also holds many prose-shaped skills that describe routing rules or judgment criteria.

The lesson is not about vendors. It is that the more a skill’s output quality matters, the less free prose it contains and the more code and numeric constraint it carries. Antigravity turning design taste into sixteen numbers and our toolkits embedding commands verbatim are the same prescription.

Separately, Gemini 3.6 Flash, the model behind this workflow, shipped on 21 July 2026 and was available in Antigravity from day one. Per Google’s announcement it uses 17 percent fewer output tokens than 3.5 Flash while output pricing dropped from $9.00 to $7.50 per million tokens. Multiplying the two effects, output cost for equivalent work lands at 0.83 times 7.50 divided by 9.00, roughly 69 percent, a saving of about 31 percent. Coding scores rose as well: DeepSWE from 37 to 49 percent, MLE Bench from 49.7 to 63.9 percent, OSWorld-Verified from 78.4 to 83.0 percent.

What this means for ThakiCloud

From the Paxis side this observation is immediately useful. Paxis is ThakiCloud’s Agent-Native Cloud control plane, treating skills, tools, policies, and audit logs as first-class resources. Its skill harness selects candidates from a large skill corpus with BM25 and executes them in an isolated sandbox. What this measurement confirms is that the selection signal is vendor-agnostic. An Antigravity skill also exposes name, description, and a “Use when” trigger clause, so to a router it is the same shape of input. Skills built in outside ecosystems can be indexed and added to the candidate pool as they are.

The boundary is equally clear. Format compatibility is not execution compatibility. This skill requires pip install opencv-python and rembg[cpu] in the user’s environment and handles local file paths directly. Pulling in arbitrary external skills uncritically drags along dependency installation and file access. That is exactly why Paxis runs skills in a sandbox and pushes every action through policy gates and audit logs. Format convergence lowers adoption cost; execution isolation remains the platform’s job.

The measurement also surfaced a task for our own corpus. Only 154 of 1911 skills, 8.1 percent, ship a bundled scripts/ directory. That signals how many procedures are still expressed as prose when deterministic code could own them, and against our internal principle of letting code own the format, there is room left.

From the ai-platform side this becomes a cost question. Agent workload cost scales with output tokens, and the 31 percent saving computed above applies when using a commercial API. An organization that cannot send code or assets outside its perimeter has to produce that same saving on premise, and the means are Kubernetes with Kueue-based GPU scheduling and vLLM serving optimization. That is the target ThakiCloud’s ai-platform aims at.

Limits and counterarguments

The largest limitation is that we did not execute anything. We never installed the Antigravity IDE and ran the skill, so whether those sixteen constraints actually produce good output is unverified. We established what the file demands, not the quality of what it yields.

The metric itself is coarse. Code fence line share is a proxy for a skill’s character, not a quality score. Skills that describe judgment criteria or routing rules should have low code share, and forcing code into them would make them worse. Reading our 18.5 percent median as a defect to fix would be a mistake.

The sample is also a single repository. One skill published by an individual developer cannot support a claim about the Antigravity skill ecosystem generally. How Google documents its official skill specification is a separate question, and we did not verify whether this repository follows it faithfully.

The Gemini 3.6 Flash figures rest on press coverage of Google’s announcement. The 17 percent token reduction is measured against the Artificial Analysis Index and varies by workload, so the 31 percent saving holds only under that premise. We did not measure it in a live application.

Finally, observing that the format has converged is not the same as standardization. Sharing name and description is a long way from compatible tool permission models, sandbox policies, and bundled asset conventions.

Wrapping up

Taking apart one published Antigravity skill, we found a SKILL.md carrying name and description with a “Use when” trigger clause exactly as ours do, 54.5 percent of its 314 lines given over to code, and design taste reduced to sixteen numeric constraints. Both claims from the opening hold. The format has converged past vendor lines, and a working skill is code and constraint rather than prompt.

There is one thing to take into practice. Every time you are tempted to write “do this well” into a new skill, ask first whether it can become a number or a line of code. If it can come down to something checkable, like 12rem or more, or between 0.88 and 0.92, write it that way. The freedom you remove from the model comes back as average output quality.

Sources

• Skill repository: WilkoMarketing/antigravity-video-websites-skill (SKILL.md, scripts/extract_frames.py)
• Gemini 3.6 Flash launch coverage: Google launches Gemini 3.6 Flash and 3.5 Flash-Lite, teases Gemini 4 (9to5Google, 21 July 2026)
• Measurement scripts and raw logs: scripts/blog/_skillmd_anatomy_20260803.py, scripts/blog/_skillmd_corpus_20260803.py, outputs/blog-impl/antigravity-skill-format-gemini-flash/run-1.log, run-2.log

One demonstration hardening into a reusable procedure. That is exactly what this tool is trying to do.

Why this matters

This post is for platform owners who hit the “who is going to write the skills” wall while rolling out agents, and for agent engineers designing a skill authoring pipeline. The conclusion first: Skill Recorder’s real contribution is not screen recording but the design decision to generalise a recording into the agent’s native tool calls rather than replaying clicks, plus opening that generalisation up so a human can review and correct it in natural language. That middle step is what separates a usable tool from a demo. Below is what we found by cloning the repository, reading the code and running its tests.

Overview

Skills as a format converged on something close to a standard over the past year. Write the procedure in a markdown file, describe in frontmatter when it applies, and the agent picks it up on its own. The problem is who writes that markdown. The person who knows a procedure best usually has no time to document it, and the person with time does not know the procedure. The bottleneck in the skill ecosystem was never consumption; it was production.

microsoft/skill-recorder aims at that bottleneck. It records a screen session while you do your normal work once, reconstructs it through the GitHub Copilot CLI as a single intent plus an ordered list of steps, and then builds a reusable skill or automation. The licence is MIT and the copyright line reads Microsoft Corporation. At the time we checked, the latest commit was 32fd0b5 dated 30 July 2026, merging release 0.3.1.

It is larger than you might guess. TypeScript-family sources including tests and docs run a little over twenty thousand lines. This is not a demo prototype; it is built to ship.

What the tool actually does

The behaviour breaks into four blocks.

First, collection. The event types defined in common/events.ts tell you immediately what it captures: session start and stop, active application switches, window title changes, clipboard changes, terminal commands, browser URLs, plus video start, stop and frame capture. Right from here you can see the posture: do not try to comprehend raw pixels, capture meaningful structured signals first.

Second, reconstruction. electron/describer/ spins up a Copilot CLI agent and has it interpret the session. The agent gets three kinds of tool. get_timeline reads the segmented timeline, get_narration reads whatever the user said out loud, and a set of frame tools pulls the screen image at a specific moment. It submits its result through submit_analysis, whose arguments are telling: a title, the intent, a confidence score for that intent, the rationale behind that confidence, and the list of steps.

Third, plan and approval. The comment block in common/skill.ts describes this precisely. Starting from an approved analysis, a multi-turn Copilot agent first proposes a plan: how it intends to generalise the recorded task, which fixed values it needs, and which native tools of the target architecture it will use. The user refines that plan in natural language, and on confirmation the final artefact is produced.

Fourth, output. There are two artefact kinds: an on-demand SKILL.md the agent invokes when its description matches the request, and an automation, meaning a multi-step procedure run on a schedule or trigger. Microsoft Scout and Microsoft 365 Copilot (Cowork) are the currently enabled targets; Copilot Studio is marked “coming soon” and greyed out in the code.

flowchart TB
    A["User performs the task once"] --> B["Collectors<br/>app switches, window titles, clipboard<br/>terminal commands, browser URLs"]
    A --> C["Optional<br/>screen video + spoken narration"]
    B --> D["events.jsonl<br/>structured events are the primary signal"]
    C --> E["Frame extraction + correlation<br/>only where events fall short"]
    D --> F["Describer agent<br/>Copilot CLI"]
    E --> F
    F --> G["submit_analysis<br/>intent, confidence, rationale, steps"]
    G --> H{"Human review"}
    H -->|"correct in natural language"| G
    H -->|"approve"| I["Builder agent proposes a plan<br/>generalisation, fixed values, native tools"]
    I --> J{"Human confirms"}
    J --> K["SKILL.md<br/>on-demand skill"]
    J --> L["Automation<br/>scheduled or triggered"]

The most impressive judgement call is how video is handled. The comments in electron/pipeline.ts state deliberately that the whole video is never scanned. Events are the primary signal; anything they fail to explain is surfaced as a probe suggestion and only harvested where confidence is low. The describer instructions repeat the same principle, telling the agent that most steps are fully explained by events alone and to budget roughly five frames. Compared with shoving an entire video into a multimodal model, this is far more realistic on both cost and accuracy.

The second standout is the rule to use intent as a filter. The instructions explicitly say not to produce a literal transcript of everything on screen. Once the intent is clear, drop the activity that does not serve it. It even singles out the first step, where the user focused the Skill Recorder window to press Start, and the last step, where they returned to press Stop, as recorder bracketing rather than user actions, and tells the agent not to emit them. If the person detoured to another site mid-demonstration, that drops out too.

Setup and integration

We pulled the repository into an isolated worktree and ran it.

git clone --depth 1 https://github.com/microsoft/skill-recorder.git
cd skill-recorder
node --version   # v24.1.0
npm --version    # 11.3.0
npm install --no-audit --no-fund

Dependency installation took 25 seconds for 609 packages, which is light for something pulling in Electron 43 and Vite 8. The runtime dependency list characterises the tool well.

"dependencies": {
  "@github/copilot-sdk": "^1.0.6",
  "@huggingface/transformers": "^4.2.0",
  "koffi": "^3.1.1",
  "sharp": "^0.34.5",
  "zod": "^4.3.6"
}

@github/copilot-sdk is the agent that performs analysis and building, koffi is the FFI used to read native window information, and sharp handles frame images. The presence of @huggingface/transformers matters: it means spoken narration is transcribed locally, through that library and the ONNX runtime. The macOS microphone usage string in the build config says the same thing, stating that the microphone is used only while narration is on and that transcription happens on this computer.

We ran the tests.

npm test
# ℹ tests 58
# ℹ pass 58
# ℹ fail 0
# ℹ duration_ms 686.846708

All 58 tests passed in 0.69 seconds. More interesting than the pass is what is being tested. The list includes items like: warn before every start until the detailed disclosure has been reviewed, the acknowledgement does not survive a new app process, session size accounts for every artefact and deleting removes the whole directory. A tool that records your screen, clipboard and terminal ought to have that minimum courtesy, and here it is nailed down in tests.

The compliance tests are equally notable. Unreviewed ONNX versions fail closed, unreviewed GitHub Copilot CLI versions fail closed, and a bundle containing only licence files cannot pass release verification. The exact items an organisation would want to check during due diligence are already automated.

One honest note. We never ran an actual recording session. It is a desktop GUI application requiring screen capture permission and a Copilot CLI login, so it cannot be reproduced end to end in an automated environment. The pipeline behaviour described above was verified by reading the code and instruction files; the numbers come from an installation and test run that genuinely happened.

What this means for ThakiCloud

Paxis, ThakiCloud’s Agent-Native Cloud, is a control plane that treats skills, tools, policies and audit logs as first-class resources. It selects candidates from more than 960 skills, executes them in isolated sandboxes, and puts every action through a policy gate and an audit log. So our interest in this tool is not “how do you choose a skill” but “where do skills come from”.

First, it names the gap on the input side precisely. In our harness, skills are either written by a person or refined from existing skills by the nightly self-evolution loop. Both start from an already documented procedure. What never enters through either path is the procedure that lives only in someone’s fingertips, such as checking a value on a particular dashboard and transferring it into a ticket template. A path from one demonstration to a draft fills that gap.

Second, the principle of generalising to native tools rather than replaying UI is worth adopting directly. Automation that retraces click coordinates breaks the moment the screen changes. Translate an observed click into the API call or CLI command that produces the same result and it lives much longer. Our own rules already say deterministic code owns format and adjudication while the model produces only content, which is the same idea in a different domain.

Third, we are borrowing the decision to ship confidence with the artefact. Attaching an intent confidence and its rationale to the analysis tells a human where to look first. During review of an auto-generated skill, that signal cuts the review cost substantially. It is a good field to add to our skill intake gate.

Fourth, the boundary is clear too. Because the artefacts target Microsoft Scout and Cowork, they do not drop straight into our harness. The intermediate artefact, however, meaning the intent and the step list, is target-neutral. Splitting there and rendering into our own format through an adapter is the realistic integration path. Conveniently, the repository keeps separate evaluation harnesses for the builder and the skill builder under evals/, so the code itself points at which boundary is replaceable.

Limits and counterarguments

The largest constraint is that this is a desktop application. A human has to sit down and demonstrate, and grant access to the screen, clipboard and terminal. It is not something you run in batch on a server. Adopting it across an organisation requires an internal policy on which screens may be recorded in the first place. It is good that the team defended this surface with tests, but tests do not substitute for the policy.

The quality of generalisation also needs verification. Deciding what is a fixed value and what varies each time, from a single demonstration, is inherently an inference. To extract “the procedure for submitting all forms” from one record of submitting one form, you have to know whether the value entered this time was an example or a constant. Get that wrong and the skill fails quietly. That is why the tool opens the plan step to the user and accepts natural language corrections, which also means the human review cannot be removed from the loop.

Finally, one demonstration does not guarantee a good skill. The way a person habitually works may not be optimal, and that inefficiency gets frozen in along with everything else. Transcribing a procedure and redesigning a procedure are different jobs. This tool is good at the former; the latter remains ours.

Wrapping up

What is worth taking from Skill Recorder is not the recording feature but three design judgements: treat structured events as the primary signal and reach for video only where confidence is low, translate observed clicks into native tool calls instead of replaying them, and open the middle of the pipeline so a human can correct intent and plan in natural language. All three transfer directly to how we design a skill authoring path.

If your team is rolling out agents, try one question. Which procedure in your organisation is repeated most often and written down by nobody? That answer is your first recording target, and the slot currently empty in your skill library.

Sources

• microsoft/skill-recorder repository (MIT): commit 32fd0b5 at time of check, release 0.3.1 (2026-07-30)
• Visual Studio Blog: Agent Skills in Visual Studio
• Microsoft Learn: Use Agent Skills with GitHub Copilot
• Original discussion: timeline post
• Execution log: installation and test figures come from outputs/blog-impl/ms-skill-recorder/run-5.log and run-6.log.

Most traffic goes to the cheap lane, a few to the premium lane. The cost structure of an agentic product is decided at that sorting step.

Why read this

This post is for engineers running an agentic product who watch the inference bill grow faster than revenue, and for infrastructure owners deciding whether to keep paying a commercial API or move workloads onto their own cluster. The short version is that the cost of an agentic product is decided not by which model you use, but by which request you send to which model. A recent founder interview shows this with unusual clarity. A product that routed every agent action to a frontier model pushed its bill to $1.2M a month, then brought it down to about $100K after two months of redesign. We look at what changed to produce a twelvefold difference, and where the same judgment lands inside our own platform.

A product whose bill broke first

The subject is a product called Polsia. Its founder, Ben Broca, runs it solo, and the goal is that a user supplies only an idea while a fleet of AI agents runs an entire company on their behalf: coding, marketing, customer acquisition, support. It drew attention when it raised $30M at a $250M valuation with zero employees, with agents even handling the data room and investor briefings. The interesting part is not the success story but the fact that the cost structure broke first.

The numbers the founder gives in the interview run like this. In March, paying users grew from 500 to 5,000, and the inference bill hit $500K in a single month, then crossed a million, peaking at $1.2M a month. The cause is simple. Every time something happened inside the product, the thing doing it was an agent, and that agent mostly called a frontier model like Opus or Sonnet. The more automation was added, and the more complex the tasks users asked for, the faster the call volume grew, more than linearly.

Here is the first lesson. Adding money does not fix the product. The founder bought himself room to absorb the cost by raising a round, but he says it plainly: cash gave breathing space, it did not solve the product. The math showed that going from 5,000 to 50,000 users would burn the entire raise within a few months, so the structure had to change before scaling. Recognizing this as an architecture problem while the bill was still affordable is the most important call in the whole story.

What changed: split the requests, build a cheap lane

The redesign started from an honest look at the traffic. In the founder’s own words, most users “ask rather simple things and have rather simple codebases, so you can standardize all of that.” The lever was the fact that requests are not uniform. A small number of hard requests and a large number of standardizable simple ones were mixed together, and processing both through the same frontier model was the real source of waste.

So the new structure is this. The standardizable long tail is pushed down to open-weight models running on rented GPUs, and only the genuinely hard reasoning stays on frontier models. When a request comes in, its complexity is classified first, and the destination model is chosen from that classification. Configuring this took about two months, and in June the bill dropped from $1.2M to roughly $100K. The twelvefold gap did not come from picking one cheaper model. It came from pulling most of the traffic out of the expensive lane in the first place.

I want to stress the second lesson here. This is not free. Those two months were real engineering: building the classifier, serving the open models, and above all verifying that the quality of requests sent to the cheaper model did not hurt the user experience. Routing without a quality-parity gate reduces cost by increasing churn. For the cheap lane to hold, you need a mechanism that decides deterministically whether “this request produces the same result on the cheaper model.”

What the classifier keys on is the hardest part of the design. Request text alone is not enough. Whether the task is generation, edit, or lookup; how large and structured the target codebase is; how deep the expected tool-call chain runs; how similar past requests were handled; these signals together decide whether a request is standardizable. And this judgment cannot be left to the model’s self-report. Not the model’s claim that “this is an easy request,” but a gate where code deterministically measures whether the output passes the same bar as the frontier model, is what earns the router trust. A classifier that quietly passes failures on the cheap lane does not reduce cost; it defers a quality incident.

flowchart TB
    A["User request<br/>(agent action)"] --> B{"Complexity<br/>classifier"}
    B -->|"Standardizable simple task<br/>(the long tail)"| C["Open-weight model<br/>owned / rented GPU"]
    B -->|"Hard reasoning<br/>(the few)"| D["Frontier model<br/>Opus / Sonnet"]
    C --> E["Quality-parity gate"]
    D --> E
    E --> F["Return response"]
    G["Agent spend & identity layer<br/>wallet · limit · metering"] -. "runaway-loop guard" .-> A

Complexity-based routing sends the standardizable many to the cheap lane and only the hard few to the premium lane. Without a quality-parity gate, the cost saving turns into churn.

Not one case, but a whole layer forming

Reading this only as one product’s war story is reading half of it. Three companies appear in the interview, and overlaying them shows how the layer that handles cost in the agent era is splitting apart.

At the top is the product layer. This is where a fleet of agents runs a 24-hour loop doing real work, as in Polsia. The defining trait of this layer is that every action is a model call, so cost explodes along with usage.

Below it is the spend and identity layer. Sapiom, which Polsia named as its agent infrastructure in the interview, sits here. Sapiom aims to give an agent a unique identity and wallet the way a person has KYC, to let it pay for external tools, APIs, and compute on a usage basis through a single API, and to stop runaway loops with spend limits and risk detection. Founded by former Shopify engineering director Ilan Zerbib, it raised a $15.75M seed according to SiliconANGLE. The moment autonomous agents buy their own resources, a mechanism that meters who spent how much at the identity level and applies limits becomes one axis of the infrastructure.

At the bottom is the serving layer. Sciforium, backed by AMD and SignalFire with roughly $12M raised, is a clear example. While most AI clouds depend on Nvidia, Sciforium puts its own high-efficiency serving stack on AMD hardware and offers multimodal inference at lower cost than the norm. With AMD engineers directly helping tune the runtime, it shows that a heterogeneous-accelerator strategy for routing around single-vendor premium is becoming real, with capital and talent behind it.

The logic running through all three companies is one thing. The moment you make a frontier model the default, the unit cost of an agentic product becomes unsustainable, and so the layers that split requests, meter spend, and push serving onto cheaper hardware are each growing large enough to be a standalone business.

What this means for ThakiCloud’s products

Overlaying this case on our own stack, we do not find an unfamiliar new demand so much as confirmation of where we already stand. This section is not about building something new; it maps where the pattern lands in our platform.

The serving and platform layer overlaps precisely with the Metis value proposition. If Polsia spent two months arriving at the conclusion “rent GPUs and serve open models behind OpenAI-compatible endpoints,” then Metis Serve and ML Studio, together with Kueue-based GPU scheduling and the model registry, are that conclusion turned into product. In particular, the heterogeneous-accelerator serving that Sciforium is trying to prove, running the same models behind the same API not only on Nvidia but on AMD or domestic NPUs, looks at the same place Metis was designed to look, as an accelerator-agnostic platform. This case is external confirmation of the market demand for the serving layer we are building.

The agent runtime layer touches the problem space of Praxis. When a fleet of agents runs continuously and every action is a model call, there are two ways to control cost: per-node complexity-based model routing, and budget caps with spend guardrails. The spend governance that Sapiom wants to hold with identity, wallet, and limits is the same kind of control layer that Praxis agents need in production. This case proves, with a bill, that a mechanism stopping an autonomous loop from calling expensive models without limit is not optional but a precondition.

To add one more note, starting on a cheaper tier by complexity and escalating to a higher model only when needed is a discipline we already follow in our own automation. Exploration and simple lookups go to lighter models, only steps that need real reasoning go to higher ones, and retro-driven escalation raises the tier only when repeated failures are observed. The conclusion Polsia reached from the outside and the routing discipline we run on the inside say the same sentence: cost is decided by which request goes where, not by model price.

Wrapping up

The most expensive mistake in building an agentic product is leaving the default that routes every action to a frontier model unexamined. Polsia’s case shows, with a bill, how that default becomes a $1.2M-a-month invoice, and how a redesign that splits requests by complexity and pushes the standardizable many onto open-weight models produces a twelvefold difference. That saving came with the price of two months of engineering and a quality-parity gate, not from picking one cheaper model. And the three layers this redesign needs, the agent runtime, spend governance, and heterogeneous serving, are splitting into a market large enough for each to be its own business, with the serving and routing layers being exactly where we already stand with Metis and Praxis.

Wide compute, one narrow pipe. That is the exact shape of an LLM decode step.

Why this matters

This post is for infrastructure owners who have to buy GPUs and put models on them, and for ML engineers deciding how hard to push their serving parameters. The conclusion first: a decode step serving a single request uses roughly 2% of the accelerator’s compute and spends the rest of its time reading weights out of memory. What you need to increase in inference serving is not GPU compute but the number of requests handled concurrently, and the threshold where that stops paying off can be computed for any device. Below we measure that threshold directly.

The topic came out of a timeline post listing what an AI engineer should learn. Near the top of that list sat two items side by side: learn the roofline model and why decode is memory-bound, and go deploy vLLM and SGLang. Those are really one story. Understand the first and the design of the second explains itself.

Overview

LLM inference splits into two phases with completely different characters. Prefill pushes the whole prompt through at once; decode produces one token at a time. Most of what a user perceives as response speed comes from decode. A 500-token answer means 500 decode steps, and the sum of those 500 is the response time.

This is where intuition breaks. A single decode step needs very little arithmetic. You push one token through each matrix multiply and you are done. Yet it is slow. The reason is not the arithmetic but the volume of weights you must read to perform it. For an 8B model in fp16 that is 16 GB pulled out of memory for every single token. The arithmetic units spend most of their time idle, waiting for data to arrive.

The tool that draws this relationship on one chart is the roofline model. Formalised by Williams, Waterman and Patterson in CACM 2009, it explains a kernel’s performance ceiling using two hardware constants: peak compute and peak memory bandwidth. Compute a kernel’s arithmetic intensity, meaning the number of floating point operations performed per byte read from memory, and you know whether it is compute-bound or bandwidth-bound.

The key result is what that number becomes for LLM decode. Take one weight matrix and push B tokens through it: the arithmetic scales with the batch, but the weight bytes you must read stay the same. In other words, the arithmetic intensity of decode is simply the batch size. At batch 1 the intensity is 1, which is dismally low on any accelerator you can buy.

What the model actually says

The roofline coordinate system is simple. Arithmetic intensity on the x-axis, achieved performance on the y-axis. The ceiling is a roof formed by two lines: a slope on the left whose gradient is the memory bandwidth, and a flat line on the right at peak compute. Where they meet is the ridge point, and its x-coordinate is peak compute divided by peak bandwidth.

Anything to the left of the ridge is bandwidth-bound. In that region, swapping in a faster arithmetic unit changes nothing. To the right of the ridge the kernel is compute-bound and widening memory does nothing. Few compasses point at the right optimisation this clearly.

flowchart TB
    A["Request arrives<br/>prompt of N tokens"] --> B["Prefill phase<br/>all N tokens in one matmul"]
    B --> C{"Arithmetic intensity<br/>= batch x sequence length"}
    C -->|"large → right of ridge"| D["Compute-bound region<br/>near the measured 5.96 TFLOP/s"]
    B --> E["KV cache populated"]
    E --> F["Decode loop<br/>one token at a time"]
    F --> G{"Arithmetic intensity<br/>= concurrent requests"}
    G -->|"small → left of ridge"| H["112 MiB of weights re-read<br/>for every token"]
    H --> I["Bandwidth-bound region<br/>saturated at the measured 124 GB/s"]
    I -.->|"continuous batching grows the batch<br/>and pushes toward the ridge"| G
    D --> J["Token streaming output"]
    I --> J

The dashed feedback edge is the essence of what serving engines like vLLM and SGLang do. The arithmetic intensity of an individual request is not set by the user, it is set by the scheduler. Group the requests in flight at the same moment into one batch and a single weight read advances all of them, raising intensity by the number you grouped. vLLM makes this practical by splitting the KV cache into fixed-size blocks that need not be physically contiguous (PagedAttention), and its chunked prefill slices long prompts so they can be interleaved between decode steps instead of monopolising an iteration. SGLang adds RadixAttention, sharing common prefixes across requests through a radix tree to cut the prefill work itself. The three techniques are not mutually exclusive and are used together.

Setup and integration

Let us measure it. All you need is PyTorch, and we used the version already present in the repository’s shared virtual environment.

# repository shared .venv (Python 3.12.8)
VIRTUAL_ENV="$PWD/.venv" uv pip install torch matplotlib
.venv/bin/python -c "import torch; print(torch.__version__, torch.backends.mps.is_available())"
# 2.13.0 True

The benchmark has three parts. First, copy a 256 MiB fp16 buffer device-to-device and derive effective bandwidth from the combined read and write traffic. Second, run a 4096 square fp16 matmul to find the compute ceiling. Third, take a 4096 x 14336 weight, the same shape as the Llama-3-8B FFN up projection, and sweep batch size from 1 to 512.

K, N = 4096, 14336          # Llama-3-8B hidden / intermediate
w = torch.empty(K, N, dtype=torch.float16, device="mps").uniform_(-0.02, 0.02)

for b in [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]:
    x = torch.empty(b, K, dtype=torch.float16, device="mps").uniform_(-1, 1)
    secs = timed(lambda: torch.matmul(x, w), "mps")   # 3 warmup + mean of 20
    flops = 2.0 * b * K * N
    total_bytes = w.numel() * 2 + (x.numel() + b * N) * 2
    # arithmetic intensity = flops / total_bytes ≈ b

It is worth pausing on why the third part is shaped like this. The arithmetic is 2 x B x K x N and the bytes read are dominated by the weight at K x N x 2. Divide one by the other and exactly B remains. This sweep is therefore a walk along the x-axis of the roofline, driven by batch size. No model checkpoint to download, no serving engine to bring up.

Everything ran inside an isolated worktree.

bash scripts/blog/impl_sandbox.sh setup llm-decode-roofline
bash scripts/blog/impl_sandbox.sh run  llm-decode-roofline -- .venv/bin/python roofline_bench.py
bash scripts/blog/impl_sandbox.sh teardown llm-decode-roofline

Measured results

The environment was macOS 26.5.2 arm64, PyTorch 2.13.0, MPS backend. Every number below is taken verbatim from run-1.log. None of it is estimated.

The two hardware constants first.

A ridge at 48 means that unless you perform at least 48 operations per byte read from memory, you cannot fill this device’s arithmetic units. And as established, the arithmetic intensity of decode equals the batch size. So until the batch reaches 48, decode is bandwidth-bound without exception.

The sweep shows exactly that.

The interval worth staring at is batch 1 through 16. Sixteen times as many tokens, and step time moved from 0.834 ms to 0.920 ms, a rise of about 10%. Fifteen of those tokens were effectively free. Reading the 112 MiB of weights dominates everything, so it barely matters how many tokens ride along on that one read.

Past batch 64 the character changes. Step time starts growing in proportion to the batch while time per token barely moves, from 22.1 µs to 19.6 µs. We have crossed the ridge into the compute-bound region. The calculated ridge of 48 sitting between the observed inflection at 32 and 64 is a clean agreement.

The measured points sit precisely on the slope. Growing the batch is not an optimisation, it is a move to the right along the axis.

Batch 1 summarised in two ratios: compute utilisation is 2.36% of peak, while effective bandwidth reads as 113.2% of the copy-based ceiling. Exceeding 100% is not an error, it is a property of the baseline. The copy benchmark includes writes, and writes carry extra traffic to fill the cache line first. A read-only matmul avoids that overhead and therefore achieves higher effective bandwidth. The conclusion is unambiguous: batch-1 decode saturates the memory bus completely while the arithmetic units idle.

The gap in per-token time between batch 1 and batch 512 is 42.6x. Same hardware, same kernel, same precision. Scheduling alone.

What this means for ThakiCloud

ThakiCloud’s ai-platform runs tenant-isolated inference workloads on Kubernetes. The measurement above compresses the reasoning we apply when sizing GPU capacity.

First, it changes how you pick a GPU. For decode-heavy workloads, memory bandwidth and capacity determine throughput far more directly than the TFLOPS figure in the catalogue. At a fixed budget there is a real region where a device with lower peak compute but wider bandwidth wins on cost per token. Which side you are on is not a matter of taste; a fifteen-line sweep answers it per device.

Second, how you partition tenants drives cost. Handing each customer a dedicated GPU is operationally simple, but it leaves every GPU running near batch 1, using a couple of percent of its arithmetic units. That is why we shard queues with Kueue and funnel requests that share a model into a single serving instance. Filling the batch toward the ridge is the same thing as reducing GPU count.

Third, it moves the break-even point for on-premise deployments. Cost per token on your own cluster is GPU hourly cost divided by actual achieved throughput. If the batch never fills, that denominator is forty times smaller and no self-hosted setup will beat a commercial API. Once traffic is dense enough for batches to fill, the cost curve of the same hardware drops sharply. Traffic density is the first thing we check with customers evaluating on-premise for regulatory or data sovereignty reasons.

Agent workloads have a convenient property here. Paxis, ThakiCloud’s Agent-Native Cloud, runs a skill harness that selects from more than 960 skills and executes them in isolated sandboxes, and that process generates many short concurrent inference requests. Unlike traffic from humans sitting one at a time in front of a chat box, agent traffic is naturally concurrent. As the curve above shows, high concurrency means operating in the low cost-per-token region. That is the advantage of running the agent platform and the inference infrastructure under one roof.

Limits and counterarguments

This was measured on a laptop-class accelerator with unified memory. Datacentre GPUs such as the H100 or H200 use HBM, with bandwidth in the single-digit TB/s range and correspondingly larger compute. The absolute numbers differ, obviously. What holds regardless of hardware is that a ridge exists and that decode’s arithmetic intensity equals batch size. In practice, compute on high-bandwidth datacentre parts has grown faster than bandwidth, so their ridge often sits further right, meaning you need an even larger batch to fill the arithmetic units.

Narrowing the measurement to a single FFN matmul is also a limitation. A real decode step includes attention kernels, and attention reads the KV cache rather than weights, so the bytes grow with sequence length. Growing the batch grows the KV cache with it, and memory capacity frequently binds before the ridge does. In production, the batch ceiling is usually set by free memory for the KV cache rather than by the roofline. That is precisely why PagedAttention concentrates on reducing fragmentation waste.

It should also be clear that growing the batch is not free. Step time at batch 512 is 10 ms against 0.834 ms at batch 1, twelve times longer. Throughput is at its best but inter-token latency for an individual user gets worse. For a conversational product, decide where you stand between throughput and latency as a service level objective first, then pick the maximum batch inside that constraint. The ridge tells you the ceiling; it does not tell you the target.

Wrapping up

Batch-1 decode uses 2.36% of the arithmetic units while completely saturating the memory bus, and growing the batch from there cuts time per token by 42.6x. Those two measured facts are the whole story. Once you know them, it becomes obvious why vLLM’s continuous batching and chunked prefill and SGLang’s prefix sharing all converged on filling the batch. They are not different optimisations; they are different ways of pushing requests rightward along the same axis.

So the next time you need to improve inference performance, add one step before you go shopping for a faster GPU. Measure bandwidth and compute on the device you intend to use, derive the ridge, and check whether your current batch sits to its left or its right. If it sits to the left, the answer is in your scheduler configuration, not in new hardware. The sweep in this post needs no model weights and no serving engine, and finishes in fifteen minutes with PyTorch alone.

Sources

• Original discussion: what an AI engineer should learn (timeline)
• Roofline: An Insightful Visual Performance Model for Multicore Architectures, Williams, Waterman and Patterson, CACM 2009
• vLLM performance and tuning docs (chunked prefill, PagedAttention)
• SGLang repository · SGLang paper (RadixAttention)
• vLLM issue #2560: compatibility of RadixAttention with existing techniques
• Measurement log: every figure comes from outputs/blog-impl/llm-decode-roofline/run-1.log.

Records that pile up every day and flow nowhere are where this paper starts.

Why this is worth reading

This is for you if you have already deployed an agent, it has not improved with time, and you are waiting on the next model release. The conclusion up front: what you are waiting for is not a better model, it is a data pipeline you have not built yet. The execution records your agent produces every day are not in a form you can learn from, and converting them into a learnable form is a systems problem, not an algorithms problem.

Overview

The paper is titled “Next-Generation Agentic Reinforcement Learning Systems Enable Self-Evolving Agents”, arXiv number 2607.01120. It went up on 1 July 2026 with a revision the next day. The authors are affiliated with Ant Group, HKUST and Tsinghua University. The title reads like another RL algorithm paper, but it is not one. It proposes no new algorithm and boasts no new benchmark score. Instead it stakes out a position.

The position is this. The foremost bottleneck for enterprise-scale self-evolving agents is not the absence of more powerful LLMs, nor the absence of more effective RL algorithms, but the absence of a system substrate that can transform deployed agent experience into governed, credit-assignable and replayable learning material.

What makes the claim interesting is that it stands in falsifiable form. If the bottleneck were the algorithm, deployed agents would improve every time a better algorithm arrived. Reality does not work that way. RL algorithms and models have both improved over the past two years, yet most agents deployed in enterprises still improve only through a human reading logs by eye, editing a prompt by hand and redeploying. The paper argues that this slow manual loop persists not out of laziness but because the plumbing does not exist.

The three gaps the paper identifies

The paper argues that current agentic RL systems and the surrounding observability stack fall short in three essential respects. These three form the spine of the argument, so it is worth taking them one at a time.

First, there is no standardized agent trajectory data protocol capable of carrying RL learning signals at step granularity across heterogeneous agent paradigms. The operative phrase is step granularity. The logs we normally keep sit at the request and response level. But when an agent produces a wrong answer after ten tool calls, the information learning needs is not that the result was wrong, it is which of the ten steps went off the rails. The claim is that no standard container exists for that information.

Second, there is no enterprise-grade comprehensive data proxy that converts real workloads into governed learning substrates. This part is closer to governance than to technology. You cannot feed execution records laced with customer data straight into training. They have to be cleaned, permission-checked and replayable when needed, and almost nobody ships that layer as a product.

Third, there is no unified agent evolution control plane that automatically decides, based on trajectory statistics, whether to update policy weights or evolve the in-context harness. This is the most operationally real of the three. When an agent underperforms you have several levers available. You can edit the prompt, add a skill, refresh memory, or retrain the model. Each differs in cost by two orders of magnitude or more. And yet the decision about which lever to pull is today made almost entirely by human intuition.

The paper proposes three co-designed pillars answering these three gaps: a standardized trajectory data protocol, an enterprise-grade data proxy, and a unified evolution control plane. It stresses that the three cannot be built separately. They have to be designed together.

How the three pillars interlock

flowchart TB
    A["Deployed agent<br/>coding assistant · support bot · research assistant"]
    A -->|"execution records accumulating daily"| P1
    subgraph S["Three pillars that must be co-designed"]
        P1["Pillar 1 · Trajectory data protocol<br/>a standard format carrying learning signal per step"]
        P2["Pillar 2 · Data proxy<br/>turns real workloads into governed learning material"]
        P3["Pillar 3 · Evolution control plane<br/>decides what to update from trajectory statistics"]
        P1 --> P2 --> P3
    end
    P3 -->|"expensive lever"| W["Retrain policy weights"]
    P3 -->|"cheap lever"| H["Prompts · skills · tools · memory"]
    W --> A
    H --> A
    P2 -.fails governance.-> X["Records unusable for learning"]

Break the chain and records pass through a dashboard and vanish. Connect it and you get a loop.

The part of this diagram worth staring at is the dotted line dropping to the lower right. If the data proxy cannot clear governance, those records cannot be used for learning. That is why the paper makes the data proxy a pillar of its own. The set of data you can technically collect and the set you may legally train on are different sets, and in an enterprise the latter is much smaller.

AReaL as the precursor implementation

The argument is not pure thought experiment. There is a real system out of the same camp. AReaL, built jointly by the RL Lab at Ant Research and Tsinghua’s Institute for Interdisciplinary Information Sciences, with the version tagged boba² billing itself as a fully asynchronous RL system.

AReaL’s core design decision is to fully decouple generation from training. In traditional synchronous RL you collect rollouts and then run training, which leaves GPUs idling in alternation. Pull the two apart asynchronously and each side can run at its own pace. The project claims roughly 2.77 times the training speed of the synchronous approach with this structure, and reported state-of-the-art results at the time on LiveCodeBench, Codeforces and CodeContests. Read those numbers with the caveat that they are the project’s own measurements.

The more practically important part sits elsewhere. AReaL lets you independently customize the dataset, the rollout behavior and the training algorithm, and drew the boundaries so that doing so does not require touching heavy system-level code. That is one of the paper’s pillars in concrete form. The format of the learning material and the execution machinery of the system have to be separable before you can evolve by swapping material. Models at 8B, 14B and 32B are published, so the structure can be opened up directly.

Holding the same ruler against our own policy file

This paper stung because the third pillar was not somebody else’s problem. ThakiCloud’s automation skills already have something control-plane shaped. A file at scripts/skills/skill_model_policy.json governs the model tier of eighteen scheduled skills, and skill_retro.py records the outcome at the end of every run and updates the policy.

The rule is simple. A skill starts on a cheap tier, gets promoted automatically once two consecutive bad runs accumulate, and the consecutive-failure counter resets on a clean run. There is no automatic demotion. In the paper’s vocabulary this is a miniature control plane deciding what to update from trajectory statistics. Of the current eighteen, ten are tier-pinned and six are running on the upper tier.

Read that file through the paper’s lens, though, and two failures come into focus. Both actually happened and both are recorded in the file.

The first failure is one of trajectory statistics quality. Several entries in the policy file carry notes that failures caused by hitting the account’s weekly usage limit were miscounted as quality failures, pushing the tier up incorrectly. Two orchestration-flavored skills were promoted that way and were manually rolled back once the cause was understood. The paper’s first gap points exactly here. If the signal is not differentiated at step granularity you cannot distinguish the kind of failure, and decisions made on statistics that cannot tell kinds apart come out wrong. Quota exhaustion and quality shortfall present as the same exit code, but the remedies are opposites.

The second failure is having only one lever. One entry in the policy file was promoted to the upper tier on 11 July 2026, and its consecutive-failure counter has climbed to twenty-one since. Promotion did not solve the problem. Because the only lever our control plane could pull was model tier, there was nothing to do once the cause lay elsewhere. That is precisely why the paper defines the control plane with weight updates and in-context harness evolution side by side. One lever does not make a control plane. It makes an auto-promotion switch.

What this means for ThakiCloud

Through the Paxis lens this paper reads like a roadmap. Paxis is the Agent-Native Cloud control plane running on top of ai-platform, treating Skills, Tools, Policies and Audit Logs as first-class resources. Map those four onto the paper’s three pillars and the picture lines up. Audit logs are where trajectory data is born, policies are the rules a data proxy applies when judging what may pass, and skills and tools are the cheap levers a control plane can pull instead of weights.

Separating what we already have from what we do not makes the next task obvious. We have audit logs, but not in a format that carries step-level learning signal. We have policy gates, but they do not connect to a path that promotes execution records into learning material. Defining skill execution logs as a replayable trajectory format is the first of the three pillars, and the other two can only be built on top of it. Knowing the order is the most practical thing this paper hands over.

Through the ai-platform lens the second pillar is an infrastructure requirement. Turning execution records into learning material means storing, cleaning and replaying those records somewhere, and when customer data is mixed in, that work becomes a question of which physical boundary the data stays inside. For customers with on-premise and sovereignty requirements this is not a negotiating point, it is a precondition. Layering trajectory collection and replay pipelines onto K8s-based multi-tenant isolation and Kueue GPU scheduling would not be selling an observability feature. It would be selling the precondition for self-evolution.

The two lenses point the same way. ai-platform physically guarantees the boundary the data stays within, and inside it Paxis enforces the policy that promotes trajectories into learning material. The paper’s insistence on co-designing the three pillars shows up again in the relationship between these two layers.

Limits and counterarguments

A few things are worth naming before taking this paper at face value.

First, this is a position paper, not a validated system report. The argument that three pillars are needed is persuasive, but there is no measurement inside the paper showing a system with all three actually achieving self-evolution at enterprise scale. There is a wide gap between an architecture proposal and a demonstration that it works, and this paper still stands on the near side.

Second, a standard protocol does not solve credit assignment for you. Building a container that holds signal per step and judging which of ten steps caused the failure are different problems. The latter is hard on its own terms and does not resolve itself once the container exists. The paper emphasizes the missing container but says relatively little about the difficulty that begins once it is full.

Third, governance costs can exceed the gains. Turning real work records into learning material means retaining customer data for a long time, and in regulated industries that is close to non-negotiable on its own. The data proxy is offered as the layer that resolves this, but in practice deciding not to use the data at all is often cheaper than building the proxy.

Fourth, automatic update decisions are themselves risky. As our two policy-file cases show, a decision made automatically on bad statistics quietly raises cost and obscures the cause. The smarter a control plane gets, the harder it becomes for a human to trace the basis of its judgment. The gain from automation and the loss in traceability have to be counted together.

Fifth, AReaL’s performance figures are self-reported by the project. That an asynchronous structure beats a synchronous one is structurally plausible, but the specific 2.77x multiple comes from the developers rather than an independent reproduction. Safer to cite it with the source attached.

Wrapping up

The value of this paper lies not in a new technique but in a relocation of the problem. It turns a gaze that looked for the reason agents fail to improve in models and algorithms toward the execution records that pile up daily and flow nowhere.

That was the conclusion stated at the top: what you are waiting for is not a better model, it is a data pipeline you have not built yet. We only confirmed, after holding the same ruler against ourselves, that the control plane we thought we had was in fact a one-lever promotion switch, and that the switch had spun twice on bad signal. Until then it looked like automation working fine.

If the agent you deployed has been unchanged for months, try one question before checking the next model release date. Where are the execution records that agent produced yesterday, and what can you change with them? If the answer ends at a dashboard, the bottleneck is not the model.

Sources

• Next-Generation Agentic Reinforcement Learning Systems Enable Self-Evolving Agents (arXiv:2607.01120)
• Revised version of the paper (arXiv:2607.01120v2)
• AReaL open-source repository (inclusionAI/AReaL)
• AReaL-boba-2-32B model card (Hugging Face)
• AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning (arXiv:2505.24298)
• Coverage of the Ant Group and Tsinghua AReaL-boba² release (DeepNewz)

Why read this

This is for infrastructure engineers who serve open-weight LLMs with vLLM or SGLang and have to answer “how many GPUs does this model need?” The conclusion first: per-token KV cache is almost independent of parameter count. It is determined by three things only, namely layer count, KV head count, and head dimension. That is why a 27B model can genuinely consume more cache per token than a 70B one.

This post does that calculation by hand. The basis is Understanding Transformers and Attention Mechanisms: An Introduction for Applied Mathematicians, posted to arXiv in April 2026. It is a 13-page introductory paper by Michel Fabrice Serret of the Paul Scherrer Institute in Switzerland, written as a presentation for the “Randomization in Transformer models” project at IPAM’s “Randomized Numerical Linear Algebra” workshop. Its arXiv classification is numerical analysis (math.NA), not machine learning.

Overview

Transformer explainers are not scarce. What makes this paper different is that it addresses applied mathematicians. Instead of building intuition through analogy, it pins down in tables what dimension each matrix has and what stays resident in memory. That is exactly the useful part for practitioners. Capacity planning reduces to “how many floats stay in memory,” and Tables 1 and 2 of the paper answer that directly.

The paper starts from tokenization and embedding, then defines attention by analogy to a database lookup. You query a database of key-value pairs and get values back, except attention returns a similarity-weighted linear combination rather than an exact match. From that base it moves to multi-head attention, and the final section covers three techniques for cutting compute and memory: KV caching, Grouped Query Attention (GQA), and latent attention (MLA). That final section is what this post digs into.

What attention actually caches

In autoregressive generation the model emits one token at a time. Every new token needs the key and value vectors of all preceding tokens, and recomputing them each step is wasteful, so they accumulate in memory. That is the KV cache. The paper notes this makes the cost of appending one token linear in token count, giving O(N_tokens² · d) overall. The price is memory: because key and value vectors must be held per layer and per head, the cost is 2 · N_L · N_h · N_KV · d multiplied by the bits per float. In the paper’s words, this “can quickly become prohibitive, especially in the long-context case.”

The first way to shrink that bottleneck is GQA. Several query heads share one key-value head, so the number of cached vectors scales with KV heads rather than query heads. The extreme case of a single KV head is Multi-Query Attention (MQA). The important consequence is that cache size is set by KV head count, not query head count. This is precisely where parameter count and cache size start to diverge.

The second is latent attention, introduced by DeepSeek. Rather than storing keys and values separately, it keeps one vector per token projected into a shared low-rank latent space, L = X·W^L. The paper describes it as “a single cache vector per token, shared between all heads.” The latent form also permits merging weight matrices: latent-to-query and latent-to-key multiply into one, and latent-to-value merges with the output weights. Fewer matrices need to be resident at inference time.

flowchart TB
    A["Input embeddings X<br/>N_KV × d_in"] --> B{"Attention variant"}
    B -->|"MHA"| C["Store K and V per head<br/>cache = N_heads × (d_QK + d_head)"]
    B -->|"GQA"| D["Query heads share KV heads<br/>cache = N_KV heads × (d_QK + d_head)"]
    B -->|"MLA"| E["Store one shared latent L = X·W^L<br/>cache = d_L"]
    C --> F["Multiply by layer count<br/>to get per-token KV cache"]
    D --> F
    E --> F
    F --> G["HBM left after weights<br/>sets concurrent token capacity"]

Tables 1 and 2 of the paper redrawn as a flow. All three branches end by multiplying through the layer count.

Porting the paper’s formulas to code

Table 1 lists the tensors multi-head attention keeps in memory; Table 2 does the same for latent attention. Extracting only the cache terms: MHA and GQA hold N_KV heads × (d_QK + d_head) floats per layer per token, while MLA holds d_L. For GQA the paper states explicitly that you replace N_heads with the KV head count in the cache terms and in W^K and W^V.

Feed in the specs from Table 3 and the numbers fall out. The calculation went into a short script.

BYTES_PER_FLOAT = 2  # fp16/bf16, the standard serving dtype

def cache_floats_per_token(m):
    """Total KV-cache floats held per token across all layers."""
    if m.kind == "mla":
        return m.layers * m.d_head          # one shared latent vector of dim d_L
    return m.layers * m.kv_heads * (m.d_head + m.d_head)   # K and V, per kv head

Rather than trusting Table 3 outright, the script cross-checks against public HuggingFace config.json files.

url = f"https://huggingface.co/{repo}/raw/main/config.json"
# compare num_hidden_layers / num_attention_heads / num_key_value_heads (or kv_lora_rank) / hidden_size

The full script lives at scripts/blog/_kvcache_math_20260803.py and the raw output at outputs/blog-impl/transformer-attention-kv-cache-math/run-1.log.

Results

Per-token KV cache for the three models, all fp16 and summed across every layer:

Computed by applying the Table 1 and Table 2 formulas to the Table 3 specs. The 128k context figure is a normalization for comparison and does not reflect each model’s actual maximum context.

The striking result is that Gemma 3 27B holds 1.55x more cache per token than Llama 3 70B, even though Llama is nearly 2.6x larger in parameters. The formula explains it. Cache is governed by layers × KV heads: Gemma 3 27B gives 62 × 16 = 992, while Llama 3 70B gives 80 × 8 = 640. Llama has more layers but cut KV heads far more aggressively, and that decision outweighed the parameter gap.

The same formula quantifies what GQA buys. Had Llama 3 70B given each of its 64 query heads its own key and value, it would need 2560 KiB per token; sharing across 8 KV heads brings that to 320 KiB, exactly an 8x reduction. The saving ratio is simply query heads divided by KV heads. Gemma 3 27B, with 32 heads over 16 KV heads, gets only 2x.

DeepSeek V2’s latent attention operates on a different level: 60 KiB per token, 8.3x less than Gemma 3 27B, despite having the most heads of the three at 128. It keeps one shared 512-dimensional latent vector instead of per-head caches.

One honest note on verification. Of the three repositories, only deepseek-ai/DeepSeek-V2 exposed its config.json without authentication, and there the values matched the paper’s Table 3 exactly: 60 layers, 128 heads, kv_lora_rank 512, hidden_size 5120. Llama 3 70B and Gemma 3 27B are gated, the script returned unreachable, and their figures come from the paper’s table as published.

What this means for ThakiCloud

This calculation maps directly onto capacity planning for ThakiCloud’s ai-platform, which schedules GPUs with Kueue on Kubernetes and serves models through vLLM. In a multi-tenant setting, how many concurrent sessions fit on one node drives unit cost, and the ceiling is set by exactly these KV cache figures.

Take a four-GPU H200 node with roughly 564GB of HBM. Llama 3 70B in fp16 occupies about 140GB of weights, leaving roughly 420GB. At 320 KiB per token that is arithmetically around 1.3 million tokens of cache, or about ten concurrent 128k-context sessions. Activation memory and page fragmentation cut into that in practice, but it fixes the order of magnitude. Put a latent-attention model on the same node and per-token cache drops by nearly an order of magnitude, raising concurrent sessions accordingly. In on-premise or sovereign deployments where you cannot simply add GPUs, that difference decides whether a deployment is viable at all.

Three practical consequences follow. First, do not estimate memory from parameter count when selecting a model; multiply num_hidden_layers by num_key_value_heads yourself. Second, the longer the context your service sells, the more KV head structure drives your cost base. Third, computing this theoretical ceiling before tuning vLLM’s gpu_memory_utilization and max_model_len narrows the search space considerably.

There is a Paxis angle too. Paxis is the Agent-Native Cloud control plane running on top of ai-platform, and its skill harness picks which model to call each turn. Agent workloads accumulate long conversation histories and tool output, so context grows quickly, which means model choice affects cache occupancy as well as token price. That a latent-attention model supports more concurrent agents on the same node is a legitimate input to routing policy.

Limits and counterarguments

These numbers are computed, not measured. They come from applying the paper’s dimension tables to published specs, not from running a benchmark, so they will differ from what vLLM actually allocates. vLLM uses PagedAttention with block-level allocation, intra-block fragmentation occurs, and prefix cache sharing or quantized KV cache shift the figures again.

The paper has its own limits. It is a 13-page workshop presentation and the author calls it a brief introduction. It proposes no new technique and runs no experiments. The three models in Table 3 are examples rather than a survey, and Llama 3, Gemma 3, and DeepSeek V2 are all several generations old as of August 2026. What remains useful is not the specific numbers but the formula, which applies unchanged to any new model’s config.json.

Concluding that latent attention always wins would also be hasty. The paper adds an important caveat: without positional encoding, latent attention can be expressed exactly as an equivalent GQA or MHA model through the low-rank factorization, but applying RoPE breaks that equivalence. RoPE is applied after the keys are constructed, and in the latent form that ordering blocks the matrix merge and adds the overhead of recomputing the positional encoding at each evaluation. Real implementations therefore append a separate “non-latent” component carrying the positional encoding, keeping the computational benefit while giving up mathematical equivalence. Cache size alone cannot decide an architecture.

Finally, KV cache is only one axis of inference memory. Weights remain the largest single block, and at small batch sizes the bottleneck shifts from capacity to bandwidth.

Wrapping up

Per-token KV cache is set by layers × KV heads × (d_QK + d_head), and parameter count never enters the equation. That is why Gemma 3 27B holds 1.55x more per token than Llama 3 70B and DeepSeek V2’s latent attention holds 8.3x less. The claim made at the top, that cache does not scale with model size, holds across all three.

Choosing your next model to serve is then straightforward. Open config.json, multiply num_hidden_layers by num_key_value_heads by head_dim, double it, and multiply by the dtype byte width. That is your per-token cache; multiply by target context length and concurrent sessions to get required HBM. It takes five minutes before you request a GPU quote, and this paper is the derivation of why those five minutes are sound.

Sources

• Paper: Understanding Transformers and Attention Mechanisms: An Introduction for Applied Mathematicians (arXiv:2604.00965, math.NA, submitted 1 April 2026, 13 pages)
• Author: Michel Fabrice Serret, Center for Scientific Computing, Theory and Data, Paul Scherrer Institute
• Cross-check: deepseek-ai/DeepSeek-V2 config.json
• Script and raw log: scripts/blog/_kvcache_math_20260803.py, outputs/blog-impl/transformer-attention-kv-cache-math/run-1.log