Research
We publish early results to workshops, but reserve finished works for production, see the blogs.
A Controlled Study of Attention-Only Transformers
Henry Ndubuaku, Karen Mosoyan, Jakub Mroz, Noah Cylich, Satyajit Kumar, Parkirat Sandhu, Roman Shemet, Justin H Lee
arXiv · 2026
Temporal Mixture-of-Experts: Serving Memory that Scales with Active Parameters
Noah Cylich, Mohsen Fayyaz, Henry Ndubuaku, Nanyun Peng
NeurIPS · 2026
How Wide Is Attention? Measuring Its Effective Width in Tokens
Karen Mosoyan, Henry Ndubuaku, Jakub Mroz, Noah Cylich, Roman Shemet, Parkirat Sandhu, Satyajit Kumar, Justin H Lee
NeurIPS · 2026
Whole-Model Quantization Below 4 Bits Per Weight for On-Device Multimodal Models
Karen Mosoyan, Henry Ndubuaku, Jakub Mroz, Noah Cylich, Roman Shemet, Parkirat Sandhu, Justin H Lee, Satyajit Kumar
NeurIPS · 2026
Parameter-Efficient Transformer Embeddings via Functional Factorization
Henry Ndubuaku
ICML · 2026
Depth Over Specialization in Small Multimodal Transformers
Jakub Mroz, Henry Ndubuaku
ICLR · 2026
Calibration-Aware Activation Sparsity for Instruction-Tuned LLMs
Noah Cylich, Karen Mosoyan, Henry Ndubuaku
ICML · 2026
A Blazing Fast LM-Head Replacement
Satyajit Kumar, Henry Ndubuaku
ICLR · 2026
Token-Aware Chunked Encoding
Parkirat Sandhu, Henry Ndubuaku, Justin Lee, Satyajit Kumar, Noah Cylich, Karen Mosoyan, Jakub Mróz
ICLR · 2026
Just Enough Learning: GRPO-Guided Controllers for Hyperparameter Sweeps
Justin H Lee, Henry Ndubuaku
ICLR · 2026
Adoption-Gated Continual Learning for On-Device Tool-Calling SLMs
Jakub Mroz, Henry Ndubuaku, Karen Mosoyan, Parkirat Sandhu, Roman Shemet
NeurIPS · 2026