K/20X LABS · AI_SETUP_FOUNDATIONS · DAILY RESEARCH BRIEF

Local AI Runtimes Advance with MLX, Vulkan, and Efficient Quantization; Sandboxes Gain Flexibility

Published , 04:44 Bogota (UTC-5) · 26 sourced items, 17 new since the previous edition · Read the foundations review · RSS

Today in 5 points

Laptop AI (MacBook, MLX)

Shared Context, Separate Engines: Stateful Streaming Inference for Multimodal LLMs on Edge SoCs

arXiv · · Laptop AI (MacBook, MLX)

A method (name not provided) enables stateful streaming inference for multimodal LLMs on edge SoCs. It allows one engine to answer from history built by another, and the CPU and GPU to decode answers to different queries simultaneously, using phase-level state

Why it matters: This improves efficiency for multimodal LLMs on edge SoCs by optimizing how different processing engines share and manage KV state, crucial for local AI on laptops and other edge d

Phone and edge AI

0.20.0-nightly.20261117NEW

LiteRT-LM releases · · Phone and edge AI

LiteRT-LM 0.20.0-nightly.20261117 updated Swift Package checksums.

Why it matters: This is a nightly update for LiteRT-LM, indicating ongoing development for phone-based AI models.

AdaFlowLite: Scalable and Non-blocking Inference on Asynchronous Mobile DataNEW

arXiv · · Phone and edge AI

AdaFlowLite is a new approach for scalable, non-blocking inference on asynchronous mobile data from multiple sensors like LiDAR and cameras. It enables opportunistic inference as partial data arrives, addressing delays or accuracy decline.

Why it matters: This matters for phone AI by enabling more efficient and responsive inference on mobile devices with diverse and dynamic sensor data, improving real-time performance.

CULypso: A Clustered Federated Unlearning paradigm for Vehicular Ad-Hoc NetworksNEW

arXiv · · Phone and edge AI

CULypso is a clustered federated unlearning paradigm for Vehicular Ad-Hoc Networks (VANETs). It proposes a hierarchical approach to integrate unlearning into edge, mobile, and time-sensitive vehicular networks, aiming to reduce communication, resource, and tim

Why it matters: This addresses privacy and resource management for AI on mobile devices in dynamic environments like VANETs, relevant for phone AI and edge computing.

Fine-Tuning Gemma 4 Edge Models for MATLAB Signal-Processing Code with SFT and QLoRA on a Laptop iGPUNEW

arXiv · · Phone and edge AI

Supervised fine-tuning (SFT) was applied to Gemma 4 edge models (E2B, E4B) for MATLAB signal-processing code. Using LoRA and QLoRA on a laptop iGPU, the pass rate increased from 18/66 to 37/66 for E2B and 32/66 to 47/66 for E4B. The 8-bit integer models were d

Why it matters: This demonstrates improved performance of small language models for specific coding tasks on local laptop hardware (iGPU) using efficient fine-tuning methods.

OAK-Mind: Zero-Training Mental Health Change Inference from Passive Sensing via Knowledge-Grounded Multi-Agent LLMsNEW

arXiv · · Phone and edge AI

OAK-Mind is a zero-training, multi-agent, knowledge-grounded LLM pipeline for inferring mental health changes from passive sensing data. It organizes sensing data into behavioral dimensions and orchestrates LLMs to reason over evidence without task-specific fi

Why it matters: This demonstrates a zero-training LLM approach for complex inference on mobile sensor data, relevant for phone AI and privacy-preserving solutions.

0.20.0-nightly.20261116

LiteRT-LM releases · · Phone and edge AI

LiteRT-LM 0.20.0-nightly.20261116 updated Swift Package checksums.

Why it matters: This is a nightly update for LiteRT-LM, indicating ongoing development for phone-based AI models.

Runtimes and quantization

b12499NEW

llama.cpp releases · · Runtimes and quantization

llama.cpp v12499 includes a Vulkan runtime loop count for Qualcomm's rms_norm. It supports macOS Apple Silicon (arm64), Intel (x64), iOS, various Linux (x64, arm64, s390x, Vulkan, CUDA, ROCm, OpenVINO, SYCL), Android (arm64, Snapdragon CPU/Adreno GPU/Hexagon N

Why it matters: This release expands hardware support for local AI inference, particularly for Qualcomm devices and various Linux/Android configurations, including NPUs.

WM-Quant: Rollout-Aware Post-Training Quantization for JEPA-Style World ModelsNEW

arXiv · · Runtimes and quantization

WM-Quant is a post-training quantization (PTQ) framework for JEPA-style world models. It addresses compounding quantization error in autoregressive model predictive control (MPC) rollouts with early-encoder precision, control-path preservation, and rollout-awa

Why it matters: This improves the efficiency of world models by reducing computational cost through quantization, which is important for local AI where resources are limited.

KVUse: Reusable-KV User Memory for Efficient LLM RecommendersNEW

arXiv · · Runtimes and quantization

KVUse is a method for efficient LLM recommenders that stores user profiles as reusable key-value (KV) cache. This avoids repeated prefill calculations of user profile text, which typically dominate serving costs for prefill-heavy ranking LLM calls.

Why it matters: This improves the efficiency of LLM-based recommenders by optimizing KV cache usage, reducing computational load for local inference of user profiles.

TempQuant: Learning Adaptive Temperatures for Quantized Language ModelsNEW

arXiv · · Runtimes and quantization

TempQuant learns adaptive temperatures for quantized language models. Quantization can degrade performance by distorting next-token probabilities and warming the model. TempQuant adjusts sampling settings to align quantized model probabilities with full-precis

Why it matters: This helps mitigate performance degradation in quantized LLMs, making them more effective for local inference where quantization is often used to reduce resource requirements.

BQ-LoRA: Binary-Quantized Low-Rank Adapters as Implicit Regularizers for Parameter-Efficient Fine-TuningNEW

arXiv · · Runtimes and quantization

BQ-LoRA introduces binary-quantized low-rank adapters for parameter-efficient fine-tuning, reducing adapter storage by 13-16x compared to QLoRA's FP16 adapters. It uses a single binary bit per adapter parameter with a learned per-layer scale and a Scaled Strai

Why it matters: This significantly reduces the memory footprint of fine-tuned models, making them more feasible for deployment on local hardware with limited resources.

v0.34.0

vLLM releases · · Runtimes and quantization

vLLM v0.34.0 introduces 10 new models, enables extensible KV cache by default for larger caches and more throughput, and removes Model Runner V1. It shows speedups on Blackwell and MI355X, up to 1.95x decode and 1.6x throughput.

Why it matters: This release significantly improves inference performance and model support for local AI, especially on high-end GPUs, by optimizing KV cache and runner architecture.

v0.40.17

Ollama releases · · Runtimes and quantization

Ollama v0.40.17 includes improvements for ChatGPT CLI restore/restart, uses thinking metadata for OpenCode settings, and uses mmap for Vulkan integrated GPUs. It also adds MLX support for concurrent generation and bumps llama.cpp and MLX/MLX-C versions.

Why it matters: This update enhances Ollama's capabilities for local AI, particularly improving performance on Vulkan integrated GPUs and adding concurrent generation support for MLX on Apple Sili

datasette-llm 0.1a11

Simon Willison · · Runtimes and quantization

Datasette-llm 0.1a11 adds support for OpenAI-compatible endpoints, allowing Datasette to use LLMs hosted anywhere that supports this protocol. This enables running Datasette Agent with local models like LM Studio, Ollama, or llama.cpp.

Why it matters: This significantly broadens the options for running local AI models with Datasette Agent by supporting a common API standard, making local inference more accessible.

Agent sandboxes (E2B and peers)

Weight-Space Programming Doesn't Need TransformersNEW

arXiv · · Agent sandboxes (E2B and peers)

Research on weight-space programming shows that LLMs can write target weights without gradient-based training. When allowed to design architectures, three frontier LLMs compiled plain-text facts into sparse tables, achieving higher test accuracy than a Transfo

Why it matters: This suggests that for weight-space programming in agent sandboxes, non-Transformer architectures designed by LLMs can be more efficient and accurate.

Can Language Models Learn to Forecast Stock Prices?

arXiv · · Agent sandboxes (E2B and peers)

BETA is a chronological sandbox for S&P 500 stock forecasting using LLMs. Frontier models in this sandbox showed weaknesses: declining scores over time, over-predicting gains, and under-predicting move sizes. Fine-tuning and reinforcement learning were applied

Why it matters: This evaluates LLMs' capabilities in a financial forecasting sandbox, highlighting current limitations and potential for improvement through fine-tuning and RL for agentic tasks.

Readie: Low-Latency Code Sandboxes for AI AgentsNEW

arXiv · · Agent sandboxes (E2B and peers)

Readie proposes learned startup snapshots for low-latency code sandboxes for AI agents. It addresses slow startup by using a set of small, quickly initialized snapshots that cover diverse agent-generated code workloads, rather than a single large snapshot.

Why it matters: This improves the efficiency and responsiveness of code sandboxes for AI agents by reducing startup latency, which is critical for iterative code generation and execution.

External sandboxes for coding agentsNEW

Simon Willison · · Agent sandboxes (E2B and peers)

Simon Willison explored external sandboxes for coding agents to enhance safety for local models. He released three sandbox server implementations: external-sandbox-v86 (Node.js/v86/WebAssembly) and external-sandbox-container (Python/Alpine Linux/Apple Containe

Why it matters: This provides practical implementations for safer local AI agent execution by isolating agent-generated code in external sandboxes, preventing damaging actions on the host machine.

datasette-apps 0.4a0

Simon Willison · · Agent sandboxes (E2B and peers)

Datasette Apps 0.4a0 adds support for linking between sandboxed HTML/JavaScript applications using URL #fragments. Apps can use these fragments, and supported URL schemes are reported to Datasette Agent for inter-app linking.

Why it matters: This enhances the functionality of sandboxed applications for AI agents, allowing for more complex and interconnected agent workflows within Datasette.

e2b@2.64.0

E2B SDK releases · · Agent sandboxes (E2B and peers)

E2B SDK 2.64.0 adds resizable templates with maxCpuCount for building and cpuCount for creating sandboxes. e2b template list now shows Max vCPUs. It also updates e2b template list -f json output and adds a shell option to commands.run.

Why it matters: This enhances the flexibility and resource management of E2B agent sandboxes by allowing customizable CPU allocation for templates and sandboxes.

Open models for local use

Solver to System: LLM Reasoning Across Representations and Execution in Scientific SimulationNEW

arXiv · · Open models for local use

Solver2System is a dataset of PDE solvers with multiple representations and controlled corruptions to evaluate LLM scientific reasoning. It tests LLMs' ability to recover physical systems through cross-representation consistency and detection of physical inval

Why it matters: This research evaluates how LLMs reason about scientific simulation code, which is relevant for agents that might generate or interpret such code in sandboxes.

RTLReviewBench: LLM Hardware Reviewers Rarely Miss Bugs but Routinely Reject Correct RTLNEW

arXiv · · Open models for local use

RTLReviewBench is a benchmark of 114 pairs of buggy and correct Verilog modules to evaluate LLM hardware reviewers. LLMs rarely accept buggy code but often reject correct code, producing more than five false rejections for every missed bug.

Why it matters: This highlights challenges for LLMs acting as agents in sandboxes when reviewing hardware code, showing a bias towards false rejections and the benefit of explicit reasoning.

Disambiguating Moral Circuits and Moral Features in Language Models: Moral content, normative judgment or cue words? Evidence from 14 modelsNEW

arXiv · · Open models for local use

Research using 227 minimal phrase pairs across 14 open-weight models found that attention heads changing moral answers belong to an evaluation circuit. This circuit responds to moral content, normative judgment, and the word "morally" in prompts.

Why it matters: This provides insight into how LLMs process moral content, which is relevant for understanding and controlling agent behavior in sandboxes.

SkillBuffBench: Measuring What Agent Skills Contribute across Harnesses, Models, Skill Access Modes and MemoryNEW

arXiv · · Open models for local use

SkillBuffBench measures the contribution of agent skills across harnesses, models, skill access modes, and memory. It evaluates how skills help coding agents, considering scenarios where agents choose from installed libraries and carry memory from previous wor

Why it matters: This benchmark provides insights into the practical utility of agent skills in sandboxes, especially concerning how agents discover and use skills with existing memory.

Method

Sources: arXiv API, Apple Machine Learning Research, NVIDIA, Google Research, Google Developers, Microsoft Research, Hugging Face, MLCommons, MIT News, Nature Machine Intelligence, Communications of the ACM, and official GitHub release feeds (MLX, llama.cpp, Ollama, vLLM, MLC LLM, LiteRT-LM, E2B). Items are filtered by topic rules; summaries are AI-assisted (gemini-2.5-flash) and grounded only in each source's own abstract or post text. Always read the linked source before acting.

Archive