K/20X LABS · AI_SETUP_FOUNDATIONS · DAILY RESEARCH BRIEF
Local AI Runtimes Advance with MLX, Vulkan, and Efficient Quantization; Sandboxes Gain Flexibility
Published , 04:44 Bogota (UTC-5) · 26 sourced items, 17 new since the previous edition · Read the foundations review · RSS
Today in 5 points
Ollama v0.40.17 adds MLX concurrent generation and Vulkan integrated GPU mmap support. llama.cpp v12499 improves Qualcomm Vulkan rms_norm and broadens hardware support. Datasette-llm 0.1a11 enables local models via OpenAI-compatible endpoints. [22][1][25]
New methods optimize LLM performance and resource use. BQ-LoRA reduces adapter storage by 13-16x for fine-tuning. TempQuant learns adaptive temperatures to mitigate quantization degradation. KVUse optimizes LLM recommenders by reusing KV cache. [9][8][5]
Agent sandboxes are gaining flexibility and security. E2B SDK 2.64.0 introduces resizable templates for CPU allocation. External sandboxes are explored for safer local agent execution, with Readie proposing learned startup snapshots for low latency. [26][19][18]
Advancements in phone AI include AdaFlowLite for non-blocking inference on asynchronous mobile data and OAK-Mind, a zero-training multi-agent LLM pipeline for mental health inference. Fine-tuning Gemma 4 edge models on laptop iGPUs also improves coding tasks. [3][15][11]
A method (name not provided) enables stateful streaming inference for multimodal LLMs on edge SoCs. It allows one engine to answer from history built by another, and the CPU and GPU to decode answers to different queries simultaneously, using phase-level state
Why it matters: This improves efficiency for multimodal LLMs on edge SoCs by optimizing how different processing engines share and manage KV state, crucial for local AI on laptops and other edge d
AdaFlowLite is a new approach for scalable, non-blocking inference on asynchronous mobile data from multiple sensors like LiDAR and cameras. It enables opportunistic inference as partial data arrives, addressing delays or accuracy decline.
Why it matters: This matters for phone AI by enabling more efficient and responsive inference on mobile devices with diverse and dynamic sensor data, improving real-time performance.
CULypso is a clustered federated unlearning paradigm for Vehicular Ad-Hoc Networks (VANETs). It proposes a hierarchical approach to integrate unlearning into edge, mobile, and time-sensitive vehicular networks, aiming to reduce communication, resource, and tim
Why it matters: This addresses privacy and resource management for AI on mobile devices in dynamic environments like VANETs, relevant for phone AI and edge computing.
Supervised fine-tuning (SFT) was applied to Gemma 4 edge models (E2B, E4B) for MATLAB signal-processing code. Using LoRA and QLoRA on a laptop iGPU, the pass rate increased from 18/66 to 37/66 for E2B and 32/66 to 47/66 for E4B. The 8-bit integer models were d
Why it matters: This demonstrates improved performance of small language models for specific coding tasks on local laptop hardware (iGPU) using efficient fine-tuning methods.
OAK-Mind is a zero-training, multi-agent, knowledge-grounded LLM pipeline for inferring mental health changes from passive sensing data. It organizes sensing data into behavioral dimensions and orchestrates LLMs to reason over evidence without task-specific fi
Why it matters: This demonstrates a zero-training LLM approach for complex inference on mobile sensor data, relevant for phone AI and privacy-preserving solutions.
llama.cpp v12499 includes a Vulkan runtime loop count for Qualcomm's rms_norm. It supports macOS Apple Silicon (arm64), Intel (x64), iOS, various Linux (x64, arm64, s390x, Vulkan, CUDA, ROCm, OpenVINO, SYCL), Android (arm64, Snapdragon CPU/Adreno GPU/Hexagon N
Why it matters: This release expands hardware support for local AI inference, particularly for Qualcomm devices and various Linux/Android configurations, including NPUs.
WM-Quant is a post-training quantization (PTQ) framework for JEPA-style world models. It addresses compounding quantization error in autoregressive model predictive control (MPC) rollouts with early-encoder precision, control-path preservation, and rollout-awa
Why it matters: This improves the efficiency of world models by reducing computational cost through quantization, which is important for local AI where resources are limited.
KVUse is a method for efficient LLM recommenders that stores user profiles as reusable key-value (KV) cache. This avoids repeated prefill calculations of user profile text, which typically dominate serving costs for prefill-heavy ranking LLM calls.
Why it matters: This improves the efficiency of LLM-based recommenders by optimizing KV cache usage, reducing computational load for local inference of user profiles.
TempQuant learns adaptive temperatures for quantized language models. Quantization can degrade performance by distorting next-token probabilities and warming the model. TempQuant adjusts sampling settings to align quantized model probabilities with full-precis
Why it matters: This helps mitigate performance degradation in quantized LLMs, making them more effective for local inference where quantization is often used to reduce resource requirements.
BQ-LoRA introduces binary-quantized low-rank adapters for parameter-efficient fine-tuning, reducing adapter storage by 13-16x compared to QLoRA's FP16 adapters. It uses a single binary bit per adapter parameter with a learned per-layer scale and a Scaled Strai
Why it matters: This significantly reduces the memory footprint of fine-tuned models, making them more feasible for deployment on local hardware with limited resources.
vLLM v0.34.0 introduces 10 new models, enables extensible KV cache by default for larger caches and more throughput, and removes Model Runner V1. It shows speedups on Blackwell and MI355X, up to 1.95x decode and 1.6x throughput.
Why it matters: This release significantly improves inference performance and model support for local AI, especially on high-end GPUs, by optimizing KV cache and runner architecture.
Ollama v0.40.17 includes improvements for ChatGPT CLI restore/restart, uses thinking metadata for OpenCode settings, and uses mmap for Vulkan integrated GPUs. It also adds MLX support for concurrent generation and bumps llama.cpp and MLX/MLX-C versions.
Why it matters: This update enhances Ollama's capabilities for local AI, particularly improving performance on Vulkan integrated GPUs and adding concurrent generation support for MLX on Apple Sili
Datasette-llm 0.1a11 adds support for OpenAI-compatible endpoints, allowing Datasette to use LLMs hosted anywhere that supports this protocol. This enables running Datasette Agent with local models like LM Studio, Ollama, or llama.cpp.
Why it matters: This significantly broadens the options for running local AI models with Datasette Agent by supporting a common API standard, making local inference more accessible.
Research on weight-space programming shows that LLMs can write target weights without gradient-based training. When allowed to design architectures, three frontier LLMs compiled plain-text facts into sparse tables, achieving higher test accuracy than a Transfo
Why it matters: This suggests that for weight-space programming in agent sandboxes, non-Transformer architectures designed by LLMs can be more efficient and accurate.
BETA is a chronological sandbox for S&P 500 stock forecasting using LLMs. Frontier models in this sandbox showed weaknesses: declining scores over time, over-predicting gains, and under-predicting move sizes. Fine-tuning and reinforcement learning were applied
Why it matters: This evaluates LLMs' capabilities in a financial forecasting sandbox, highlighting current limitations and potential for improvement through fine-tuning and RL for agentic tasks.
Readie proposes learned startup snapshots for low-latency code sandboxes for AI agents. It addresses slow startup by using a set of small, quickly initialized snapshots that cover diverse agent-generated code workloads, rather than a single large snapshot.
Why it matters: This improves the efficiency and responsiveness of code sandboxes for AI agents by reducing startup latency, which is critical for iterative code generation and execution.
Simon Willison · · Agent sandboxes (E2B and peers)
Simon Willison explored external sandboxes for coding agents to enhance safety for local models. He released three sandbox server implementations: external-sandbox-v86 (Node.js/v86/WebAssembly) and external-sandbox-container (Python/Alpine Linux/Apple Containe
Why it matters: This provides practical implementations for safer local AI agent execution by isolating agent-generated code in external sandboxes, preventing damaging actions on the host machine.
Simon Willison · · Agent sandboxes (E2B and peers)
Datasette Apps 0.4a0 adds support for linking between sandboxed HTML/JavaScript applications using URL #fragments. Apps can use these fragments, and supported URL schemes are reported to Datasette Agent for inter-app linking.
Why it matters: This enhances the functionality of sandboxed applications for AI agents, allowing for more complex and interconnected agent workflows within Datasette.
E2B SDK releases · · Agent sandboxes (E2B and peers)
E2B SDK 2.64.0 adds resizable templates with maxCpuCount for building and cpuCount for creating sandboxes. e2b template list now shows Max vCPUs. It also updates e2b template list -f json output and adds a shell option to commands.run.
Why it matters: This enhances the flexibility and resource management of E2B agent sandboxes by allowing customizable CPU allocation for templates and sandboxes.
Solver2System is a dataset of PDE solvers with multiple representations and controlled corruptions to evaluate LLM scientific reasoning. It tests LLMs' ability to recover physical systems through cross-representation consistency and detection of physical inval
Why it matters: This research evaluates how LLMs reason about scientific simulation code, which is relevant for agents that might generate or interpret such code in sandboxes.
RTLReviewBench is a benchmark of 114 pairs of buggy and correct Verilog modules to evaluate LLM hardware reviewers. LLMs rarely accept buggy code but often reject correct code, producing more than five false rejections for every missed bug.
Why it matters: This highlights challenges for LLMs acting as agents in sandboxes when reviewing hardware code, showing a bias towards false rejections and the benefit of explicit reasoning.
Research using 227 minimal phrase pairs across 14 open-weight models found that attention heads changing moral answers belong to an evaluation circuit. This circuit responds to moral content, normative judgment, and the word "morally" in prompts.
Why it matters: This provides insight into how LLMs process moral content, which is relevant for understanding and controlling agent behavior in sandboxes.
SkillBuffBench measures the contribution of agent skills across harnesses, models, skill access modes, and memory. It evaluates how skills help coding agents, considering scenarios where agents choose from installed libraries and carry memory from previous wor
Why it matters: This benchmark provides insights into the practical utility of agent skills in sandboxes, especially concerning how agents discover and use skills with existing memory.
Hugging Face and OpenRouter have partnered to facilitate running open-weight models.
Why it matters: This partnership aims to make open-weight models more accessible for local deployment and use, benefiting local AI enthusiasts and developers.
Method
Sources: arXiv API, Apple Machine Learning Research, NVIDIA, Google Research, Google Developers, Microsoft Research, Hugging Face, MLCommons, MIT News, Nature Machine Intelligence, Communications of the ACM, and official GitHub release feeds (MLX, llama.cpp, Ollama, vLLM, MLC LLM, LiteRT-LM, E2B). Items are filtered by topic rules; summaries are AI-assisted (gemini-2.5-flash) and grounded only in each source's own abstract or post text. Always read the linked source before acting.