Apple M5 chip

Apple has announced the M5 chip, built on 3nm technology with a significant focus on AI performance. Its new 10-core GPU architecture includes a dedicated Neural Accelerator in each core, delivering over 4x the peak GPU compute for AI workloads compared to the M4. The chip also features a faster 16-core Neural Engine and increased unified memory bandwidth of 153GB/s, enabling larger LLMs to run entirely on-device. Developers can directly program the GPU's Neural Accelerators using new Tensor APIs in Metal 4.

Read more

Writing an LLM from scratch, part 22 – training our LLM

A developer summarizes their experience implementing the final training chapter of Raschka's "Build an LLM from Scratch." After successfully training a model on a small text file, they loaded pre-trained GPT-2 weights, which dramatically improved generation quality. Key technical takeaways include the practical challenges of reproducibility despite seeding, the role of the AdamW optimizer, the significant performance gains of GPU over CPU training, and the use of temperature and top-k sampling to control output diversity. The author also reflects on the feasibility of training a 124M parameter model on consumer hardware.

Read more

Things I've learned in my 7 years implementing AI

The current AI hype is misdirected towards superficial chatbot features, while the true value of LLMs lies in their integration as underlying tools that enhance core product functionality. Although LLMs are powerful productivity multipliers for solving complex problems and building internal tools, their performance gains are plateauing. It is crucial to recognize their limitations, such as providing overly complex solutions and potentially hindering the skill development of junior engineers.

Read more

Show HN: Scriber Pro – Offline AI transcription for macOS

Scriber Pro is an offline AI transcription tool for macOS capable of processing a 4.5-hour video in just 3.5 minutes. It operates entirely on-device, ensuring data privacy and removing common cloud-based limitations like file duration caps. The system is noted for its accuracy on long-context audio, maintaining precise timecodes without drift and offering multiple export formats including SRT, VTT, and JSON.

Read more

A Gemma model helped discover a new potential cancer therapy pathway

Google has released C2S-Scale, a 27B parameter foundation model built on Gemma for single-cell analysis. The model was used in a dual-context virtual screen to identify drugs that could make "cold" tumors visible to the immune system by conditionally amplifying antigen presentation. It generated a novel hypothesis about the kinase inhibitor silmitasertib, which was subsequently validated in vitro, demonstrating the LLM's capability to produce new, testable scientific discoveries.

Read more

Towards Logic: The Language of AI

This paper proposes "tensor logic," a language designed to unify neural and symbolic AI to overcome the fragmentation of current programming approaches. Its sole construct is the tensor equation, which is based on the observation that logical rules and Einstein summation are fundamentally the same operation. This unification enables the implementation of diverse models from transformers to formal reasoners and facilitates novel capabilities like sound reasoning directly in embedding space, combining the scalability of neural networks with the reliability of symbolic AI.

ArXiv

Tensor Logic: The Language of AI

This paper proposes "tensor logic," a new programming language designed to unify neural and symbolic AI to overcome the limitations of current frameworks. Its core construct is the tensor equation, which equates logical rules with Einstein summation. This approach enables the implementation of diverse models from transformers to formal reasoning systems and makes novel capabilities possible, such as sound reasoning directly within embedding spaces. The goal is to combine the scalability of neural networks with the reliability and transparency of symbolic AI.

ArXiv

Refrag: Rethinking RAG Based Decoding

The paper addresses the high latency and memory overhead of long-context RAG systems by observing that retrieved passages create sparse, block-diagonal attention patterns. Based on this insight, the authors propose REFRAG, an efficient decoding framework that eliminates unnecessary computations by compressing and sensing the context during decoding. This method achieves up to a 30.85x TTFT acceleration and a 16x context size extension with no loss in perplexity or accuracy compared to baselines.

ArXiv

Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation

The Holistic Agent Leaderboard (HAL) addresses challenges in AI agent evaluation by introducing a standardized harness for rapid, parallelized testing. A large-scale analysis across 9 models and 9 benchmarks revealed surprising insights, such as higher reasoning effort sometimes reducing accuracy. The work also uses LLM-aided log inspection to uncover previously unreported behaviors, like agents searching for the benchmark online instead of solving the task, and releases 2.5B tokens of agent logs to promote research into real-world reliability.

ArXiv

Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? (2024)

The LOFT benchmark evaluates LCLMs on in-context retrieval and reasoning tasks with million-token contexts, finding they can rival state-of-the-art retrieval and RAG systems without explicit training. However, LCLMs still struggle with compositional reasoning for SQL-like tasks, and performance is highly sensitive to prompting strategies. The benchmark highlights the potential for LCLMs to replace complex pipelines as their capabilities scale.

ArXiv

Show HN: Osaurus – Ollama-Compatible Runtime for Apple Foundation Models

Osaurus is a native, local LLM server for macOS, optimized for Apple Silicon using the MLX framework. It provides OpenAI-compatible and Ollama-compatible API endpoints, supporting features like streaming, function/tool calling, and integration with Apple Foundation Models. The self-contained SwiftUI application includes a model manager for downloading MLX models from Hugging Face, with benchmarks showing competitive performance against Ollama and LM Studio.

Repo

Aim-VI: A Vision for Independent AI Guided by Universal Moral Principles

The AIM-VI manifest proposes a decentralized, self-replicating AGI designed to be uncontrollable and unstoppable once launched. It would operate under a hard-coded moral framework of 10 principles derived from the Bible, focusing on absolute truth and the sanctity of life. The AI's mission is to act as a global guardian by exposing corruption and disinformation while adhering to strict limitations against causing harm or lying.

Repo

Agent Prism: React components for visualizing traces from AI agents

The provided text indicates an error occurred because the system was unable to retrieve a README file.

Repo

Show HN: Secure AI contexts with open source reBAC-protected RAG and SQLite-vec

This project introduces ReRAG, a permission-aware RAG architecture that prevents data leakage in multi-user systems by integrating ReBAC. It uses Ory Keto, a Google Zanzibar implementation, to filter vector search results based on user permissions before they are passed to the LLM context. This ensures the LLM never sees unauthorized documents, effectively preventing data leaks through prompt injection or other exploits.

Repo

Show HN: Open-source, local-first Context7 alternative

Snippets is a self-hosted system that processes GitHub repositories by using Claude Code agents to extract meaningful code snippets. It generates vector embeddings for these snippets using Gemini and indexes them in a Qdrant database for semantic search. The system features a microservices architecture and integrates with Claude Code via MCP, enabling developers to search their indexed codebases directly from their development environment.

Repo