K/20X LABS · AI_SETUP_FOUNDATIONS · DAILY RESEARCH BRIEF

Local AI Runtimes Advance, On-Device Models Gain Multimodal Support, and Agent Sandboxes Enhance Security

Published , 04:44 Bogota (UTC-5) · 20 sourced items, 10 new since the previous edition · Read the foundations review · RSS

Today in 5 points

Phone and edge AI

EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware AbilitiesNEW

arXiv · · Phone and edge AI

EngineAgent is an agentic system that automates on-device LLM model enablement by separating engine and hardware knowledge from model-specific implementation. It constructs a hierarchy-aware engine index.

Why it matters: This system addresses the challenge of supporting new LLMs on diverse mobile hardware, streamlining on-device inference engine development.

Domain-Invariant and Language-Model-Assisted Acoustic Side-Channel Attacks on KeyboardsNEW

arXiv · · Phone and edge AI

This paper shows that acoustic side-channel attacks can recover typed text from keystrokes in open-world conditions by using a domain-invariant acoustic model and a language model for correction.

Why it matters: This highlights a security vulnerability related to local device interaction, where AI can be used for malicious purposes to infer user input.

Yonkers Rises

Simon Willison · · Phone and edge AI

"Yonkers Rises" is a tool that uses Claude Fable 5.2 Medium to generate a three.js animation of Yonkers as a mech, using OpenStreetMap data.

Why it matters: This shows an application of generative AI for 3D modeling and animation, potentially relevant for agent sandboxes or local creative AI tools.

v0.19.1

LiteRT-LM releases · · Phone and edge AI

LiteRT-LM v0.19.1 adds support for multimodal (vision + audio) Gemma 4 models on the Artisan GPU backend and enables bidirectional attention for vision processing on GPU Artisan models.

Why it matters: This update expands LiteRT-LM's capabilities for multimodal AI on mobile GPUs, enhancing on-device processing for Gemma 4 models.

From Chips to Skills, NVIDIA and Korea Expand Sovereign AI Collaboration

NVIDIA Blog · · Phone and edge AI

NVIDIA and Korea's MSIT are expanding collaboration across sovereign AI ecosystems, including AI semiconductors, data center infrastructure, talent, startups, models, agents, and security.

Why it matters: This collaboration focuses on developing AI hardware and ecosystems, potentially impacting the availability and capabilities of NPUs/XPUs for local and edge AI.

Runtimes and quantization

v0.34.0rc2NEW

vLLM releases · · Runtimes and quantization

vLLM v0.34.0rc2 stops setting gpu_memory_utilization in most extensible contexts.

Why it matters: This runtime change affects how GPU memory is managed, which can impact local AI deployments.

b12317NEW

llama.cpp releases · · Runtimes and quantization

llama.cpp release b12317 adds a bin kernel for OpenCL, specifically `kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_`.

Why it matters: This update to llama.cpp enhances OpenCL support, potentially improving performance for local inference on compatible hardware.

v0.34.0rc1NEW

vLLM releases · · Runtimes and quantization

vLLM v0.34.0rc1 includes a bugfix for greedy_argmax in Spec Decode and XPU, treating NaN as -inf on the top.

Why it matters: This bugfix improves the stability and correctness of greedy decoding in vLLM, relevant for local inference accuracy.

v0.40.17

Ollama releases · · Runtimes and quantization

Ollama v0.40.17 improves ChatGPT CLI restore, adds graceful restart, uses mmap for Vulkan integrated GPUs, and supports concurrent generation on MLX. It also bumps llama.cpp and MLX versions.

Why it matters: This Ollama update brings performance and stability improvements, including concurrent generation for MLX, beneficial for local AI on Apple silicon.

v0.40.16

Ollama releases · · Runtimes and quantization

Ollama v0.40.16 introduces Ripple Preview, a 65B dense model for agentic coding. It also reduces memory usage for Qwen3.8 Flash Next on Apple silicon and updates llama.cpp.

Why it matters: This Ollama release offers a new agentic coding model and significant memory optimization for Apple silicon, improving local AI performance.

MINE: Entropy-coded Memory-efficient Inference for Large Language Models

arXiv · · Runtimes and quantization

MINE (Memory-efficient INference Engine) is a lossless entropy coding approach that accelerates single-stream inference of LLMs by exploiting non-uniform weight distribution.

Why it matters: MINE provides a method to accelerate memory-bound single-stream LLM inference, which is crucial for efficient local deployments on consumer hardware.

Agent sandboxes (E2B and peers)

How Artificial Analysis Puts Frontier Models to the Test with E2B

E2B Blog · · Agent sandboxes (E2B and peers)

Artificial Analysis uses E2B to run agentic benchmarks on public and pre-release models, scaling from zero to thousands of tasks quickly.

Why it matters: This demonstrates E2B's utility as an agent sandbox for evaluating and benchmarking AI models efficiently.

NVIDIA Vera brings faster compute to E2B sandboxes

E2B Blog · · Agent sandboxes (E2B and peers)

NVIDIA Vera provides faster compute and memory population for E2B sandboxes, with capacity for 1,550 loaded sandboxes on one server.

Why it matters: This enhances E2B's agent cloud capabilities by offering faster and higher-capacity compute resources for running agent sandboxes.

Open models for local use

The Wizard of Open Weights: Chris Alexiuk Makes AI Easy With Open Models

NVIDIA Blog · · Open models for local use

Chris Alexiuk, a senior product research engineer at NVIDIA, advocates for open source AI and making open LLMs practical for developers.

Why it matters: This highlights the importance of open models for local AI development and accessibility, aligning with the K/20X LABS focus.

Method

Sources: arXiv API, Apple Machine Learning Research, NVIDIA, Google Research, Google Developers, Microsoft Research, Hugging Face, MLCommons, MIT News, Nature Machine Intelligence, Communications of the ACM, and official GitHub release feeds (MLX, llama.cpp, Ollama, vLLM, MLC LLM, LiteRT-LM, E2B). Items are filtered by topic rules; summaries are AI-assisted (gemini-2.5-flash) and grounded only in each source's own abstract or post text. Always read the linked source before acting.

Archive