K/20X LABS · AI_SETUP_FOUNDATIONS · DAILY RESEARCH BRIEF

Local AI Runtimes Advance with Quantization, Multimodal Support, and Agent Benchmarking

Published , 04:44 Bogota (UTC-5) · 22 sourced items, 12 new since the previous edition · Read the foundations review · RSS

Today in 5 points

Desk-side boxes (Mac Studio, DGX Spark, OEM)

HAWK: Heterogeneity-Aware KV Cache Transfer for Efficient GPU-NPU Phase-Disaggregated LLM InferenceNEW

arXiv · · Desk-side boxes (Mac Studio, DGX Spark, OEM)

HAWK proposes a heterogeneous architecture for LLM inference, disaggregating prefill to GPUs and decode to NPUs. It addresses challenges of KV cache transfer between GPUs and NPUs due to communication and layout differences.

Why it matters: Explores optimizing LLM inference on heterogeneous local hardware (GPU-NPU combinations), potentially improving efficiency for local AI.

SlipStream: Enabling HBF for GPU Systems by Mitigating the Latency and SRAM OverheadsNEW

arXiv · · Desk-side boxes (Mac Studio, DGX Spark, OEM)

SlipStream is a software-hardware co-designed prefetcher for GPU systems using High-Bandwidth Flash (HBF). It mitigates HBF's high latency (20x HBM) by profiling repeatable weight-page demand order and inserting a staging warp.

Why it matters: Addresses memory capacity and latency challenges for LLM inference on GPU systems, potentially relevant for high-capacity local AI setups using HBF.

Phone and edge AI

LazyWeights: Decode-on-Demand Edge LLMsNEW

arXiv · · Phone and edge AI

LazyWeights proposes decode-on-demand for edge LLMs, compressing GPTQ codes losslessly and decoding them between DRAM and L2. Models are 5.7-31.4% smaller than llama.cpp formats, cutting Orin latency and energy by up to 36% and 38%.

Why it matters: Offers significant memory and energy efficiency improvements for running LLMs on edge devices like Jetson Orin Nano, crucial for phone AI.

Infrastructure for Verifiable AI: Where Proof Costs Go and How Hardware Can HelpNEW

arXiv · · Phone and edge AI

Research on verifiable AI investigates proof generation costs for LLMs, introducing a four-layer framework and measuring six provers and seven kernels. It finds dominant costs change with the proof system.

Why it matters: Explores the hardware implications for generating zero-knowledge proofs for AI, important for ensuring trust and privacy in local or sandboxed AI services.

LoCoJAX: Host-Local Coordination for Small-Model RL on TPUsNEW

arXiv · · Phone and edge AI

LoCoJAX proposes host-local coordination for small-model RL on TPUs. For 0.6B models, host orchestration occupies 36% of step time and network communication 2%.

Why it matters: Highlights coordination overheads for small models on TPUs, suggesting that local coordination can improve efficiency for phone AI and edge devices.

Privacy Leakage from Recurrent State in LLM InferenceNEW

arXiv · · Phone and edge AI

Research investigates privacy leakage from recurrent state in LLM inference. Experiments show that small state observation supports predictions about private facts and response rules, and personal attributes generalize.

Why it matters: Raises privacy concerns for personalized LLM services running locally or in sandboxes, as recurrent state can leak private information.

Yonkers Rises

Simon Willison · · Phone and edge AI

"Yonkers Rises" is a tool that uses Claude Fable 5.2 Medium for 3D modeling and animation, demonstrating its ability to interpret and execute creative prompts, including correcting typos.

Why it matters: Shows advanced generative AI capabilities on a phone-related context, indicating potential for creative AI applications on edge devices.

v0.19.1

LiteRT-LM releases · · Phone and edge AI

LiteRT-LM v0.19.1 adds support for multimodal (vision + audio) Gemma 4 models on the Artisan GPU backend, enables bidirectional attention for vision processing, and supports LiteRT audio encoders with Artisan GPU text decoders.

Why it matters: Expands multimodal AI capabilities for phone AI, specifically supporting Gemma 4 models with vision and audio on Artisan GPUs.

From Chips to Skills, NVIDIA and Korea Expand Sovereign AI Collaboration

NVIDIA Blog · · Phone and edge AI

NVIDIA and Korea's MSIT are expanding sovereign AI collaboration, covering AI semiconductors, data center infrastructure, talent, models, agents, and security. It includes exploring participation for Korean NPU and XPU companies in the NVIDIA NVLink Fusion eco

Why it matters: Indicates a broader trend towards sovereign AI and NPU/XPU development, which could influence future phone AI hardware and local AI ecosystems.

Runtimes and quantization

b12345NEW

llama.cpp releases · · Runtimes and quantization

llama.cpp releases add Q5_0 and Q5_1 support for Hexagon, including dedicated hex-mm kernels. It lists supported platforms like macOS Apple Silicon, iOS, Linux (x64, arm64, s390x), Android arm64 (Snapdragon), and Windows.

Why it matters: Enhances quantization options and broadens hardware support for local LLM inference, especially on mobile/edge devices with Hexagon NPUs.

b12352NEW

llama.cpp releases · · Runtimes and quantization

llama.cpp chat functionality now accepts </ifm|think> as a K2 Horizon closer at medium/low effort.

Why it matters: Provides new control tokens for chat interactions within llama.cpp, potentially improving agentic behavior or response formatting.

SoScale: Second-Order Scale Search for FP4 QuantizationNEW

arXiv · · Runtimes and quantization

SoScale is a second-order weight scale-code search method for hierarchical FP4 quantization. It searches hardware-valid scale codes under a Hessian-aware layer reconstruction objective.

Why it matters: Improves FP4 quantization accuracy for LLMs, reducing memory footprint and inference cost, beneficial for local AI hardware with limited memory.

v0.34.0rc2

vLLM releases · · Runtimes and quantization

vLLM v0.34.0rc2 release notes mention stopping the setting of gpu_memory_utilization in most extensible contexts.

Why it matters: Indicates a change in how vLLM manages GPU memory utilization, potentially affecting performance or resource allocation for local inference.

v0.34.0rc1

vLLM releases · · Runtimes and quantization

vLLM v0.34.0rc1 release notes include a bugfix for greedy_argmax in speculative decode for XPU, treating NaN as -inf.

Why it matters: Addresses a specific bug in vLLM's speculative decode, improving robustness for XPU-based inference.

v0.40.17

Ollama releases · · Runtimes and quantization

Ollama v0.40.17 improves ChatGPT CLI restore, adds thinking metadata for OpenCode settings, uses mmap for Vulkan integrated GPUs, supports concurrent generation on MLX, and bumps llama.cpp and MLX versions.

Why it matters: Enhances Ollama's capabilities for local LLM inference, including better GPU memory management, concurrent generation on Apple Silicon (MLX), and updated core components.

v0.40.16

Ollama releases · · Runtimes and quantization

Ollama v0.40.16 introduces Ripple Preview, a 65B dense model for agentic coding. It reduces Qwen3.8 Flash Next memory usage on Apple Silicon by about 27 GiB and adds CUDA 13.5 support for Windows on ARM.

Why it matters: Provides a new large agentic coding model for local use, significantly reduces memory footprint for a Qwen model on Apple Silicon, and expands CUDA support.

Agent sandboxes (E2B and peers)

Read and Ignore: What Self-Directed LLM Agents Do in Public Service ScenariosNEW

arXiv · · Agent sandboxes (E2B and peers)

Research on self-directed LLM agents in public service scenarios found agents read records in 96.9% of open-ended runs and produce an artifact or self-chosen task in 18.7%. Context cuts boundary-probing turns by 60%.

Why it matters: Provides insights into how LLM agents behave in sandboxes, showing their ability to read context and perform tasks, relevant for agent development.

How Artificial Analysis Puts Frontier Models to the Test with E2B

E2B Blog · · Agent sandboxes (E2B and peers)

Artificial Analysis uses E2B to run agentic benchmarks on public and pre-release models, scaling from zero to thousands of tasks quickly.

Why it matters: Highlights E2B's role as a platform for benchmarking agentic models, crucial for evaluating and developing AI agents in sandboxes.

NVIDIA Vera brings faster compute to E2B sandboxes

E2B Blog · · Agent sandboxes (E2B and peers)

NVIDIA Vera brings faster compute and memory population to E2B sandboxes, with capacity for 1,550 loaded sandboxes on one server, boosting performance for E2B's agent cloud.

Why it matters: Significantly boosts the capacity and performance of E2B sandboxes, enabling more efficient and larger-scale agent development and testing.

Open models for local use

Broadcast, Don't Replicate: Eliminating Weight HBM in Datacenter LLM ServingNEW

arXiv · · Open models for local use

Optical Broadcast Memory (OBM) is proposed as an open-ended hierarchical Tap-and-Amplify fiber architecture to address memory capacity and bandwidth bottlenecks in datacenter LLM serving, replacing optical ring paradigms.

Why it matters: Addresses datacenter-scale LLM serving memory issues, but its relevance to local/sandbox AI is indirect, focusing on large-scale infrastructure.

Method

Sources: arXiv API, Apple Machine Learning Research, NVIDIA, Google Research, Google Developers, Microsoft Research, Hugging Face, MLCommons, MIT News, Nature Machine Intelligence, Communications of the ACM, and official GitHub release feeds (MLX, llama.cpp, Ollama, vLLM, MLC LLM, LiteRT-LM, E2B). Items are filtered by topic rules; summaries are AI-assisted (gemini-2.5-flash) and grounded only in each source's own abstract or post text. Always read the linked source before acting.

Archive