Ludwig - Resources

2024-01-01–ongoing

resources: 1497

tags: 252

last sync: 2026-05-28 16:01

This is a collection of browser bookmarks and YouTube videos I have been gathering since the 1st of January 2024. Each resource is classified within a Wikipedia-inspired hierarchical tag system in an attempt to capture and organize my primary areas of interest.

The tag tree is kept up-to-date using a pipeline using Claude Code and Codex. Going down one level in the tag tree implies you've zoomed in on a narrower domain in a given field. This is supposed to help keep the whole thing cohesive.

Quality varies semi-intentionally. There are many highly-qualitative resources but you'll also find more casual reads. The only moment of "curation" is me deciding this was interesting enough to bookmark.

To help with discoverability, you can find a fuzzy search bar. Each tag in tree also has a dedicated page.

About this site Export as JSON

ai (370)
- compression (15)
- computer_vision (15)
- deep_learning (59)
  - fundamentals (42)
  - implementations (11)
  - recommender_systems (1)
  - theory (5)
- ethics (13)
- game_ai (8)
- generative_models (30)
  - diffusion (30)
- interpretability (45)
- language_models (155)
  - applications (9)
    
    agents (7)
    
    multimodal (2)
  - architectures (74)
    
    context (3)
    
    transformers (71)
    
    attention_mechanisms (70)
    
    inference_caching (1)
  - efficiency (38)
  - evaluation (5)
  - inference (3)
  - model_reports (5)
  - theory (7)
  - training (9)
  - usage (5)
- nlp (15)
  - embeddings (13)
  - machine_translation (1)
  - speech (1)
- reinforcement_learning (15)
  - fundamentals (6)
  - llm (6)
  - policy_methods (3)
biology (7)
- computational_biology (7)
biomedical_engineering (6)
- bioelectronics (5)
- microelectrode_arrays (1)
cognition (97)
- consciousness (22)
- decision_making (11)
- learning (28)
- learning_methods (6)
- memory (3)
- perception (2)
- productivity (25)
cs (1183)
- algorithms (2)
  - dynamic_programming (2)
- computer_architecture (138)
  - cpu_design (30)
  - hardware (108)
    
    ai_accelerators (29)
    
    arm (4)
    
    floating_point (5)
    
    gpus (31)
    
    memory_models (15)
    
    optimization (16)
    
    vectorization (8)
- computer_graphics (28)
  - 3d_math (4)
  - rendering (18)
  - webgl (6)
- courses (4)
- cryptography (6)
- databases (11)
- distributed_systems (10)
- game_development (26)
  - graphics_programming (13)
  - physics_simulation (8)
  - procedural_generation (5)
- hardware_engineering (13)
- history (25)
  - hardware (15)
  - people (10)
- networking (5)
- parallel_computing (28)
  - cuda (24)
  - simd (4)
- programming_languages (149)
  - c (33)
  - cpp (4)
    
    stl (4)
  - haskell (13)
  - jai (3)
  - odin (2)
  - python (7)
  - rust (42)
  - typescript (1)
  - zig (44)
- signal_processing (3)
- software_architecture (1)
  - ffi (1)
- software_development (267)
  - architecture (17)
  - build_systems (1)
    
    nix (1)
  - burnout (2)
  - concurrency (21)
    
    asynchronous_programming (7)
    
    atomics (14)
  - data_oriented_design (1)
    
    key_value_stores (1)
  - data_structures (7)
    
    hash_maps (4)
  - debugging (11)
  - design (25)
    
    command_line (3)
    
    interfaces (6)
    
    robustness (5)
    
    simplicity (9)
  - devops (6)
  - educational_resources (10)
  - engineering_culture (2)
  - ide (6)
    
    architectures (6)
  - memory (28)
    
    safety (28)
  - performance_optimization (84)
  - security (10)
    
    application_security (10)
  - sustainability (6)
  - testing (4)
    
    integration_testing (1)
  - text_editors (4)
    
    vim (4)
  - text_processing (4)
  - tutorials (8)
  - user_interfaces (10)
- systems_programming (141)
  - assembly (9)
  - compilers (16)
  - debugging (2)
  - distributed_systems (28)
  - emulators (7)
  - executables (1)
    
    pe_format (1)
  - filesystem_correctness (2)
  - io_uring (1)
  - latency (9)
  - linkers (11)
  - memory_management (4)
    
    arena_allocators (3)
  - networking (4)
  - operating_systems (42)
  - signals (1)
  - text_editors (2)
  - virtualization (2)
- theory (320)
  - algorithms (25)
    
    boolean_satisfiability (5)
    
    compression (3)
    
    dynamic_programming (1)
    
    hash_functions (1)
    
    matrix_multiplication (6)
    
    minimization (2)
    
    parallel (1)
    
    verification (6)
  - compilers (113)
    
    analysis (2)
    
    code_generation (19)
    
    history (1)
    
    intermediate_representation (17)
    
    jit (6)
    
    llvm (14)
    
    optimization (24)
    
    parsing (13)
    
    research (1)
    
    specialized_crypto (1)
    
    toolchains (4)
    
    type_systems (11)
  - computation (27)
    
    complexity (8)
    
    quantum (2)
    
    models (19)
  - data_structures (10)
  - formal_verification (22)
    
    proof_assistants (22)
  - hypercomputation (6)
  - networks (4)
  - programming_languages (74)
    
    design (45)
    
    functional (26)
    
    metaprogramming (1)
    
    rust (1)
    
    zig (1)
  - quantum_computing (3)
  - systems_programming (1)
    
    design (1)
  - type_theory (35)
- tools (5)
  - build_systems (1)
  - neovim (1)
  - terminal_emulators (2)
  - window_managers (1)
- web_technologies (1)
  - wasm (1)
economics (27)
- macroeconomics (1)
- semiconductors (26)
mathematics (293)
- algebraic_geometry (8)
- algebraic_topology (7)
- analysis (9)
- category_theory (68)
- education (4)
- entropy (7)
- geometry (1)
  - celestial_mechanics (1)
- group_theory (9)
- history (15)
- information_theory (36)
- learning_resources (10)
- linear_algebra (25)
- logic (65)
- measure_theory (4)
- non_euclidean_geometry (2)
- numerical_analysis (2)
- statistics (16)
- topology (5)
philosophy (113)
- aesthetics (2)
- epistemology (37)
- ethics (3)
- history (3)
- logic (24)
- metaphysics (17)
- mind (25)
- political (2)
physics (46)
- classical_mechanics (3)
- continuum_mechanics (2)
- cosmology (3)
- electromagnetism (1)
- fluid_dynamics (3)
- nanotechnology (1)
- particle_physics (2)
- quantum_gravity (3)
- quantum_mechanics (3)
- statistical_physics (19)
- theoretical_foundations (6)
technical_writing (12)
- documentation (3)
- programming_guides (9)
uncategorized (7)

Recent Bookmarks

Haskell is not category theory

bookmark · Added on May 28, 2026 · 17 min read

What, why, who?

A Geometric Calculator Inside a Neural Network

bookmark · Added on May 28, 2026 · 5 min read

We found a neural mechanism that operates over manifolds: a general-purpose addition module inside Llama 3.1 8B which manipulates circular representations of numbers.

Topological constraints on self-organization in locally interacting systems | Philosophical Transactions of the Royal Society A | The Royal Society

bookmark · Added on May 28, 2026 · 50 min read

Francesco Sacco, Dalton Sakthivadivel, Michael Levin; Topological constraints on self-organization in locally interacting systems. Philos Trans A Math Phys Eng Sci 14 May 2026; 384 (2320): 20250011.

The Real Singularity is the Friends We Made Along the Way

bookmark · Added on May 28, 2026 · 4 min read

This was in the Financial Times. I don’t have a subscription, so I don’t know what article it was in, or what context, or if they are in on the joke of the general absurdity of the graph or not.

Topological Data Analysis as a Morphometric Method: Using Persistent Homology to Demarcate a Leaf Morphospace

bookmark · Added on May 28, 2026 · 33 min read

Current morphometric methods that comprehensively measure shape cannot compare the disparate leaf shapes found in seed plants and are sensitive to processing...

A New Consciousness of Mathematics

bookmark · Added on May 28, 2026 · 42 min read

In an age of proof abundance, what becomes scarce?

MAST90068

bookmark · Added on May 28, 2026 · 5 min read

MAST90068

MAST30026

bookmark · Added on May 28, 2026 · 11 min read

MAST30026

Beginner's Guide to Linkers

bookmark · Added on May 28, 2026 · 43 min read

This article is intended to help C & C++ programmers understand the essentials of what the linker does.

Dispatches from the possibly last days of human relevance

bookmark · Added on May 28, 2026 · 6 min read

As most readers have presumably heard by now, Paul Erdös’s Unit Distance Problem from 1946—one of the central open problems from the field of discrete geometry—has been solved by …

See all bookmarks →

Recent Videos

Introduction to Homological Algebra II: Basic Notions & Examples

video · Added on May 8, 2026

In this talk, I introduce the primary definition(s) of the course, along with some "low dimensional" examples from graph theory.

Journée ImpAct 2026 - David Bessis

video · Added on May 8, 2026

🎤 Conférence : "Les maths à l'ère de l'IA" de David Bessis, lors de la "Journée Impact : Enjeux Sociétaux de l'IA" du 8 février 2026 ! Co-financé par l'institu...

"On the Mechanics of Cellular and Multicellular Active Matter" by Haiqian Yang

video · Added on April 16, 2026

This is a ~53 minute talk + Q&A titled "On the Mechanics of Cellular and Multicellular Active Matter" by Haiqian Yang (https://www.linkedin.com/in/haiqian-yang-...

How Attention Residuals Rewire Modern LLMs

video · Added on April 16, 2026

Attention Residuals replaces the standard fixed residual accumulation with softmax attention over previous layer outputs. This enables each layer to selectively...

Polynomial Functors Course, Day 1: Introduction

video · Added on April 16, 2026

Course website: https://topos.site/poly-course Book: https://topos.site/poly-book.pdf Book suggestions document: https://docs.google.com/document/d/1qY5hLglgSW...

David Jaz Myers - Tutorial on Lenses

video · Added on April 16, 2026

Tutorial given at Applied Category Theory 2023 https://act2023tutorials.netlify.app/ https://act2023.github.io/

David Jaz Myers: Paradigms of composition

video · Added on April 16, 2026

MIT Category Theory Seminar 2020/10/01 ©Spifong Speaker: David Jaz Myers Title: Paradigms of composition Abstract: Scientists and engineers manage the blist...

Bourbaki, les années 1945-75 - Jean-Pierre Serre, Pierre Cartier, Jacques Dixmier & Alain Connes

video · Added on April 16, 2026

Découvrez l'histoire fascinante du groupe Bourbaki, ce collectif de mathématiciens qui a révolutionné la discipline au XXe siècle. Dans cette rencontre exceptio...

Henri Cartan une vie de mathématicien

video · Added on April 16, 2026

Programming with Categories - Lecture 1

video · Added on March 26, 2026

Video lectures at MIT. See http://brendanfong.com/programmingcats.html Lecturers: Brendan Fong, Bartosz Milewski, David Spivak Summary: In this course we expl...

See all videos →

Domain specific architectures for AI inference

bookmark · Added on May 29, 2025 · 32 min read

fleetwood.dev

DeepSeek-V3 Explained 1: Multi-head Latent Attention

bookmark · Added on May 29, 2025 · 9 min read

Key architecture innovation behind DeepSeek-V2 and DeepSeek-V3 for faster inference

Optimizing Transformer-Based Diffusion Models for Video Generation with NVIDIA TensorRT

bookmark · Added on May 16, 2025 · 6 min read

State-of-the-art image diffusion models take tens of seconds to process a single image. This makes video diffusion even more challenging, requiring significant computational resources and high costs.

You could have designed state of the art positional encoding

bookmark · Added on May 16, 2025 · 13 min read

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

attention is logarithmic, actually

bookmark · Added on May 16, 2025 · 11 min read

supaiku dot com § attention is logarithmic, actually § time complexity is a very bad model when working with parallelism. in which i make the case for work-depth analysis instead of time complexity.

AI Arrives In The Middle East: US Strikes A Deal with UAE and KSA – SemiAnalysis

bookmark · Added on May 16, 2025 · 14 min read

The US has signed two landmark agreements with the United Arab Emirates and Kingdom of Saudi Arabia (KSA) that that will noticeably shift the balance of power. The deals have economic, geopolitical…

Transformers Represent Belief State Geometry in their Residual Stream

bookmark · Added on May 16, 2025 · 19 min read

Produced while being an affiliate at PIBBSS[1]. The work was done initially with funding from a Lightspeed Grant, and then continued while at PIBBSS.…

Llama from scratch (or how to implement a paper without crying)

bookmark · Added on May 16, 2025 · 11 min read

I want to provide some tips from my experience implementing a paper. I'm going to cover my tips so far from implementing a dramatically scaled-down versio...

The Curse of Knowing How, or; Fixing Everything

bookmark · Added on May 16, 2025 · 9 min read

A reflection on control, burnout, and the strange weight of technical fluency.

High-Performance Domain-Specific Compilation Without Domain-Specific Compilers

bookmark · Added on May 16, 2025 · 4h 59m read

The MAP-Elites Algorithm: Finding Optimality Through Diversity

bookmark · Added on May 16, 2025 · 1 min read

MAP-Elites is a method in reinforcement learning to avoid the local optimum of a search space by storing multiple candidate solutions…

How To Scale

bookmark · Added on May 13, 2025 · 53 min read

While there are already excellent posts on scaling, I wanted to share my own understanding and things i've learned from my past few months and hopefully spark some discussion. I hope this post can shed light for anyone navigating the challenges of scaling up neural networks. And there may be mistakes or inaccuracies, so if you want to correct me or would like to discuss further, please feel free to DM me on X or leave a comment.

Deep Dive into Yann LeCun’s JEPA

bookmark · Added on May 6, 2025 · 29 min read

ML blog.

Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?

bookmark · Added on May 4, 2025 · 50 min read

As AI systems are used in high-stakes applications, ensuring interpretability is crucial. Mechanistic Interpretability (MI) aims to reverse-engineer neural networks by extracting human-understandable algorithms to explain their behavior. This work examines a key question: for a given behavior, and under MI's criteria, does a unique explanation exist? Drawing on identifiability in statistics, where parameters are uniquely inferred under specific assumptions, we explore the identifiability of MI explanations. We identify two main MI strategies: (1) "where-then-what," which isolates a circuit replicating model behavior before interpreting it, and (2) "what-then-where," which starts with candidate algorithms and searches for neural activation subspaces implementing them, using causal alignment. We test both strategies on Boolean functions and small multi-layer perceptrons, fully enumerating candidate explanations. Our experiments reveal systematic non-identifiability: multiple circuits can replicate behavior, a circuit can have multiple interpretations, several algorithms can align with the network, and one algorithm can align with different subspaces. Is uniqueness necessary? A pragmatic approach may require only predictive and manipulability standards. If uniqueness is essential for understanding, stricter criteria may be needed. We also reference the inner interpretability framework, which validates explanations through multiple criteria. This work contributes to defining explanation standards in AI.

Are Transformers universal approximators of sequence-to-sequence functions?

bookmark · Added on May 3, 2025 · 1 min read

Despite the widespread adoption of Transformer models for NLP tasks, the expressive power of these models is not well-understood. In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, which is quite surprising given the amount of shared parameters in these models. Furthermore, using positional encodings, we circumvent the restriction of permutation equivariance, and show that Transformer models can universally approximate arbitrary continuous sequence-to-sequence functions on a compact domain. Interestingly, our proof techniques clearly highlight the different roles of the self-attention and the feed-forward layers in Transformers. In particular, we prove that fixed width self-attention layers can compute contextual mappings of the input sequences, playing a key role in the universal approximation property of Transformers. Based on this insight from our analysis, we consider other simpler alternatives to self-attention layers and empirically evaluate them.

a Hugging Face Space by nanotron

bookmark · Added on May 3, 2025 · 1 min read

The ultimate guide to training LLM on large GPU Clusters

arXiv:quant-ph/0011122v2 20 Dec 2000

bookmark · Added on May 3, 2025 · 1h 44m read

An elementary proof of a universal approximation theorem

bookmark · Added on May 1, 2025 · 10 min read

In this short note, we give an elementary proof of a universal approximation theorem for neural networks with three hidden layers and increasing, continuous, bounded activation function. The result is weaker than the best known results, but the proof is elementary in the sense that no machinery beyond undergraduate analysis is used.

A Group and Its Center, Intuitively

bookmark · Added on April 27, 2025 · 3 min read

Last week we took an intuitive peek into the First Isomorphism Theorem as one example in our ongoing discussion on quotient groups.

Understanding Entanglement With SVD

bookmark · Added on April 27, 2025 · 9 min read

Quantum entanglement is, as you know, a phrase that's jam-packed with meaning in physics. But what you might not know is that the linear algebra behind it is quite simple.

Training Large Language Models to Reason in a Continuous Latent Space

bookmark · Added on April 24, 2025 · 50 min read

Large language models (LLMs) are restricted to reason in the “language space”, where they typically express the reasoning process with a chain-of-thought (CoT) to solve a complex reasoning problem.

A Guide on Semiconductor Development

bookmark · Added on April 24, 2025 · 16 min read

How It's Made: Fancy Sand

Recommended Books and Resources

bookmark · Added on April 24, 2025 · 5 min read

Across all experience levels.

The Computer as a Communication Device

bookmark · Added on April 24, 2025 · 34 min read

ARM's Chernobyl Moment

bookmark · Added on April 22, 2025 · 6 min read

Incredibly self-destructive cancelation of Qualcomm's v8 ALA.

The Uses of Complacency

bookmark · Added on April 22, 2025 · 13 min read

The optimal amount of self-satisfaction is not zero

The Book of Shaders

bookmark · Added on April 22, 2025 · 2 min read

Gentle step-by-step guide through the abstract and complex universe of Fragment Shaders.

Unstructured Thoughts on the Problems of OSS/FOSS

bookmark · Added on April 22, 2025 · 10 min read

Originally from replies to a Twitter thread: https://x.com/TheGingerBill/status/1914389352416993395 This is not a structured argument against FOSS/OSS but my uncommon thoughts on the topic. I am not sure if I agree [that FOSS/OSS derives from the same thinking process as the ideology of communism], but I understand the sentiment. The fundamental issue is that software is trivially copyable. I have loads of issues with FOSS and OSS1. And part of this “ideology” (as presented in the original post) is naïvety coupled with only first-order thinking and a poor understanding of ownership.

On Bloat

bookmark · Added on April 22, 2025 · 1 min read

On Bloat Rob Pike Brighter Tech Commonwealth Bank September 30, 2024

Training Large Language Models to Reason in a Continuous Latent Space

bookmark · Added on April 22, 2025 · 1 min read

Large language models (LLMs) are restricted to reason in the "language space", where they typically express the reasoning process with a chain-of-thought (CoT) to solve a complex reasoning problem. However, we argue that language space may not always be optimal for reasoning. For example, most word tokens are primarily for textual coherence and not essential for reasoning, while some critical tokens require complex planning and pose huge challenges to LLMs. To explore the potential of LLM reasoning in an unrestricted latent space instead of using natural language, we introduce a new paradigm Coconut (Chain of Continuous Thought). We utilize the last hidden state of the LLM as a representation of the reasoning state (termed "continuous thought"). Rather than decoding this into a word token, we feed it back to the LLM as the subsequent input embedding directly in the continuous space. Experiments show that Coconut can effectively augment the LLM on several reasoning tasks. This novel latent reasoning paradigm leads to emergent advanced reasoning patterns: the continuous thought can encode multiple alternative next reasoning steps, allowing the model to perform a breadth-first search (BFS) to solve the problem, rather than prematurely committing to a single deterministic path like CoT. Coconut outperforms CoT in certain logical reasoning tasks that require substantial backtracking during planning, with fewer thinking tokens during inference. These findings demonstrate the promise of latent reasoning and offer valuable insights for future research.

On the Biology of a Large Language Model

bookmark · Added on April 22, 2025 · 3h 34m read

Large language models display impressive capabilities. However, for the most part, the mechanisms by which they do so are unknown.

Do Llamas Work in English? On the Latent Language of Multilingual Transformers

bookmark · Added on April 22, 2025 · 1 min read

We ask whether multilingual language models trained on unbalanced, English-dominated corpora use English as an internal pivot language -- a question of key importance for understanding how language models function and the origins of linguistic bias. Focusing on the Llama-2 family of transformer models, our study uses carefully constructed non-English prompts with a unique correct single-token continuation. From layer to layer, transformers gradually map an input embedding of the final prompt token to an output embedding from which next-token probabilities are computed. Tracking intermediate embeddings through their high-dimensional space reveals three distinct phases, whereby intermediate embeddings (1) start far away from output token embeddings; (2) already allow for decoding a semantically correct next token in the middle layers, but give higher probability to its version in English than in the input language; (3) finally move into an input-language-specific region of the embedding space. We cast these results into a conceptual model where the three phases operate in "input space", "concept space", and "output space", respectively. Crucially, our evidence suggests that the abstract "concept space" lies closer to English than to other languages, which may have important consequences regarding the biases held by multilingual language models.

The Unsustainability of Moore’s Law

bookmark · Added on April 22, 2025 · 15 min read

Roughly every two years, the density of transistors that can be fit onto a silicon chip doubles.

The Era of Experience Paper

bookmark · Added on April 22, 2025 · 31 min read

"Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?"

bookmark · Added on April 22, 2025 · 3 min read

This isn't a new intuition, but a nice new set of results.

tt-metal/tech_reports/memory/allocator.md at main · tenstorrent/tt-metal

bookmark · Added on April 19, 2025 · 5 min read

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model. - tenstorrent/tt-metal

What Modern NVMe Storage Can Do, And How To Exploit It: High-Performance I/O for High-Performance Storage Engines

bookmark · Added on April 19, 2025 · 55 min read

Memory on Tenstorrent

bookmark · Added on April 19, 2025 · 14 min read

When I first started programming Metalium. Memory was a

Multi-layer language heads: the output latent is for text (and nothing else)

bookmark · Added on April 19, 2025 · 4 min read

The last layer’s hidden state in a transformer is meant only for being decoded into token probabilities. Don’t use it for autoregressive image generation Dont’t use it for looped latent transformers Only use it to produce the next token in a language model It is a compressed representation of the...

Subnanosecond flash memory enabled by 2D-enhanced hot-carrier injection

bookmark · Added on April 19, 2025 · 36 min read

A two-dimensional Dirac graphene-channel flash memory based on a two-dimensional-enhanced hot-carrier-injection mechanism that supports both electron and hole injection is used to make devices with a subnanosecond program speed.

Andrew_S._Tanenbaum_-_Structured_Computer_Organization

bookmark · Added on April 19, 2025 · 25h 42m read

CS336: Language Modeling from Scratch

bookmark · Added on April 19, 2025 · 4 min read

Language models serve as the cornerstone of modern natural language processing (NLP) applications and open up a new paradigm of having a single general purpose system address a range of downstream tasks.

A Gentle Introduction to Lambda Calculus - Part 1: Syntax

bookmark · Added on April 19, 2025 · 7 min read

Even though lots of people nowadays advocate for applying functional programming principles to JavaScript, not many of them know the principles of Lambda Cal...

Getting Started

bookmark · Added on April 19, 2025 · 19 min read

Yet it seems to me that the situation right now is that LtU has readers with very different backgrounds, among them many readers who haven't studied PL formally.

curry-howard.dvi

bookmark · Added on April 19, 2025 · 6h 46m read

Intelligence as efficient model building

bookmark · Added on April 19, 2025 · 45 min read

Personal site for posts about my interests: the biotech industry, medicine, molecular biology, neuroscience, biorisk, science, consciousness, AI, innovation, decision making, philosophy, games, sci-fi, probability, and forecasting (among other things). I write to learn, mostly about biotech.

Contextualization Machines

bookmark · Added on April 17, 2025 · 17 min read

Astro description

What Is ChatGPT Doing … and Why Does It Work?

bookmark · Added on April 15, 2025 · 1h 33m read

Stephen Wolfram explores the broader picture of what's going on inside ChatGPT and why it produces meaningful text. Discusses models, training neural nets, embeddings, tokens, transformers, language syntax.

Mission Apollo: Landing Optical Circuit Switching at Datacenter Scale

bookmark · Added on April 10, 2025 · 44 min read

In this paper, we describe Apollo, to the best of our knowledge, the world's first large-scale production deployment of optical circuit switches (OCSes) for datacenter networking. We will first describe the infrastructure challenges and use cases that motivated optical switching inside datacenters. We then delve into the requirements of OCSes for datacenter applications: balancing cost, port count, switching time, and optical performance, which drive design choices and implementation details of our internally developed 3D MEMS-based OCS. To enable the Apollo optical switching layer, we employ circulators to realize bidirectional links through the OCS, effectively doubling the OCS radix. The OCS and circulator design choices were critical for meeting network bandwidth, scale, and cost targets. We review the critical co-design of WDM transceiver technology for these OCS plus circulator-based bidirectional links and their corresponding physical impairments, delivered over four generations/speeds of optical interconnect. Finally, we conclude with thoughts on future directions in hardware development and associated applications.

Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes

bookmark · Added on April 10, 2025 · 1h 5m read

I argue that data becomes temporarily interesting by itself to some self-improving, but computationally limited, subjective observer once he learns to predict or compress the data in a better way, thus making it subjectively simpler and more beautiful. Curiosity is the desire to create or discover more non-random, non-arbitrary, regular data that is novel and surprising not in the traditional sense of Boltzmann and Shannon but in the sense that it allows for compression progress because its regularity was not yet known. This drive maximizes interestingness, the first derivative of subjective beauty or compressibility, that is, the steepness of the learning curve. It motivates exploring infants, pure mathematicians, composers, artists, dancers, comedians, yourself, and (since 1990) artificial systems.

Position: Model Collapse Does Not Mean What You Think

bookmark · Added on April 10, 2025 · 1 min read

The proliferation of AI-generated content online has fueled concerns over \emph{model collapse}, a degradation in future generative models' performance when trained on synthetic data generated by earlier models. Industry leaders, premier research journals and popular science publications alike have prophesied catastrophic societal consequences stemming from model collapse. In this position piece, we contend this widespread narrative fundamentally misunderstands the scientific evidence. We highlight that research on model collapse actually encompasses eight distinct and at times conflicting definitions of model collapse, and argue that inconsistent terminology within and between papers has hindered building a comprehensive understanding of model collapse. To assess how significantly different interpretations of model collapse threaten future generative models, we posit what we believe are realistic conditions for studying model collapse and then conduct a rigorous assessment of the literature's methodologies through this lens. While we leave room for reasonable disagreement, our analysis of research studies, weighted by how faithfully each study matches real-world conditions, leads us to conclude that certain predicted claims of model collapse rely on assumptions and conditions that poorly match real-world conditions, and in fact several prominent collapse scenarios are readily avoidable. Altogether, this position paper argues that model collapse has been warped from a nuanced multifaceted consideration into an oversimplified threat, and that the evidence suggests specific harms more likely under society's current trajectory have received disproportionately less attention.

paper

bookmark · Added on April 10, 2025 · 2h 37m read

88_HC2024.Tenstorrent.Jasmina.Davor.v7

bookmark · Added on April 10, 2025 · 6 min read

RWKV Language Model

bookmark · Added on April 7, 2025 · 1 min read

The RWKV Language Model

Recent AI model progress feels mostly like

bookmark · Added on April 7, 2025 · 11 min read

About nine months ago, I and three friends decided that AI had gotten good enough to monitor large codebases autonomously for security problems. We s…

Device Placement Optimization with Reinforcement Learning

bookmark · Added on April 6, 2025 · 31 min read

The past few years have witnessed a growth in size and computational requirements for training and inference with neural networks. Currently, a common approach to address these requirements is to use a heterogeneous distributed environment with a mixture of hardware devices such as CPUs and GPUs. Importantly, the decision of placing parts of the neural models on devices is often made by human experts based on simple heuristics and intuitions. In this paper, we propose a method which learns to optimize device placement for TensorFlow computational graphs. Key to our method is the use of a sequence-to-sequence model to predict which subsets of operations in a TensorFlow graph should run on which of the available devices. The execution time of the predicted placements is then used as the reward signal to optimize the parameters of the sequence-to-sequence model. Our main result is that on Inception-V3 for ImageNet classification, and on RNN LSTM, for language modeling and neural machine translation, our model finds non-trivial device placements that outperform hand-crafted heuristics and traditional algorithmic methods.

Building an Open Future

bookmark · Added on April 5, 2025 · 10 min read

We are building an open future for AI. Own your silicon future. Join us.

diffusion transformers

bookmark · Added on April 5, 2025 · 1 min read

Metaphorically, you can think of Vision Transformers as the eyes of the system, able to understand and contextualize what it sees, while Stable Diffusion is the hand of the system, able to generate and manipulate images based on this understanding.

Faking ADTs and GADTs in Languages That Shouldn't Have Them

bookmark · Added on April 1, 2025 · 34 min read

Haskell is the world’s best programming language, but let’s face the harsh reality that a lot of times in life you’ll have to write in other programming languages. But alas you have been fully Haskell-brained and lost all ability to program unless it is type-directed, you don’t even know how to start writing a program without imagining its shape as a type first. Well, fear not. The foundational theory behind Algebraic Data Types and Generalized Algebraic Data Types (ADTs and GADTs) are so fundamental that they’ll fit (somewhat) seamlessly into whatever language you’re forced to write. After all, if they can fit profunctor optics in Microsoft’s Java code, the sky’s the limit! This is an “April Fools” joke in the tradition of my previous one in some of these ways that we are going to twist these other languages might seem unconventional or possibly ill-advised… but also the title is definitely a lie: these languages definitely should have them! :D

Accelerate

bookmark · Added on March 29, 2025 · 1 min read

Accelerate is a language for array-based computations, designed to exploit massive parallelism.

Ok Rust, You Really Have a Readability Problem

bookmark · Added on March 29, 2025 · 1 min read

Rust is safe. Rust is fast. Rust is powerful. And Rust is… sometimes completely unreadable.

Circuit Tracing: Revealing Computational Graphs in Language Models

bookmark · Added on March 29, 2025 · 3h 43m read

Deep learning models produce their outputs using a series of transformations distributed across many computational units (artificial “neurons”).

Things that go wrong with disk IO

bookmark · Added on March 29, 2025 · 5 min read

Things that go wrong with disk IO

Analyzing Modern NVIDIA GPU cores

bookmark · Added on March 29, 2025 · 1 min read

GPUs are the most popular platform for accelerating HPC workloads, such as artificial intelligence and science simulations. However, most microarchitectural research in academia relies on GPU core pipeline designs based on architectures that are more than 15 years old. This paper reverse engineers modern NVIDIA GPU cores, unveiling many key aspects of its design and explaining how GPUs leverage hardware-compiler techniques where the compiler guides hardware during execution. In particular, it reveals how the issue logic works including the policy of the issue scheduler, the structure of the register file and its associated cache, and multiple features of the memory pipeline. Moreover, it analyses how a simple instruction prefetcher based on a stream buffer fits well with modern NVIDIA GPUs and is likely to be used. Furthermore, we investigate the impact of the register file cache and the number of register file read ports on both simulation accuracy and performance. By modeling all these new discovered microarchitectural details, we achieve 18.24% lower mean absolute percentage error (MAPE) in execution cycles than previous state-of-the-art simulators, resulting in an average of 13.98% MAPE with respect to real hardware (NVIDIA RTX A6000). Also, we demonstrate that this new model stands for other NVIDIA architectures, such as Turing. Finally, we show that the software-based dependence management mechanism included in modern NVIDIA GPUs outperforms a hardware mechanism based on scoreboards in terms of performance and area.

Advanced Performance Optimizations for Models

bookmark · Added on March 29, 2025 · 26 min read

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model. - tenstorrent/tt-metal

paper.dvi

bookmark · Added on March 29, 2025 · 35 min read

þÿKevin-and-Nick.PDF

bookmark · Added on March 29, 2025 · 36 min read

Move Slow and Fix Things

bookmark · Added on March 28, 2025 · 6 min read

Growing up as a kid in rural Bavaria, I always …

Why is Yazi fast?

bookmark · Added on March 28, 2025 · 6 min read

This article assumes that you have already used Yazi and are familiar with most of its features.

User Guide for NVPTX Back-end

bookmark · Added on March 28, 2025 · 46 min read

To support GPU programming, the NVPTX back-end supports a subset of LLVM IR along with a defined set of conventions used to represent GPU programming concepts.

An AnandTech Interview with Jim Keller: 'The Laziest Person at Tesla'

bookmark · Added on March 27, 2025 · 1h 15m read

I've spoken about Jim Keller many times on AnandTech.

Notes/Primer on Clang Compiler Frontend (1) : Introduction and Architecture

bookmark · Added on March 25, 2025 · 10 min read

Notes/Primer on Clang Compiler Frontend: Introduction and Architecture These are my notes on chapters 1 & 2 of the Clang Compiler Frontend by Ivan Murashko. The book is focused on teaching the fundamentals of LLVM to C++ engineers who are interested in learning about compilers to optimize their daily workflow by enhancing their code quality and overall development process. (I’ve referened this book extensively, and a lot of the snippets here are from this book.

Implementation of simple microprocessor using verilog

bookmark · Added on March 25, 2025 · 14 min read

I am trying to make a simple microprocessor in verilog as a way to understand verilog and assembly at the same time. I am not sure if I am implementing what I think of microprocessors well enough ...

learn-fpga/FemtoRV/TUTORIALS/FROM_BLINKER_TO_RISCV/README.md at master · BrunoLevy/learn-fpga · GitHub

bookmark · Added on March 24, 2025 · 1h 45m read

Learning FPGA, yosys, nextpnr, and RISC-V . Contribute to BrunoLevy/learn-fpga development by creating an account on GitHub.

Why async Rust?

bookmark · Added on March 24, 2025 · 30 min read

I genuinely can’t understand how anybody could look at the mess that’s Rust’s async and think that it was a good design for a language that already had the reputation of being very complicated to write.

Softmax Attention is a Fluke

bookmark · Added on March 24, 2025 · 13 min read

Calibrated AttentionCalibrated Attention NanoGPTAttention is the magic ingredient of modern neural networks. It is the core of what has launched performant language models into the spotlight starting with GPT, and since then, it has extended its hands across all modalities.There are a number of desirable properties that make attention a first-class building block. Namely: • It handles variable sequence lengths with ease • It allows for a global receptive field without needing to scale parameters

Transformers Laid Out

bookmark · Added on March 23, 2025 · 38 min read

I have encountered that there are mainly three types of blogs/videos/tutorials talking about transformers

Template Haskell

bookmark · Added on March 22, 2025 · 21 min read

Intuitively Template Haskell provides new language features that allow us to convert back and forth between concrete syntax, i. e.

A friendly introduction to machine learning compilers and optimizers

bookmark · Added on March 18, 2025 · 21 min read

[Twitter thread, Hacker News discussion]

Comments on Source

bookmark · Added on March 18, 2025 · 4 min read

The section of the wiki allows anyone to document, explain, post questions, or make comments on the Lua source code. You may link to [1] or paste the code in question.

The_Implementation_of_Lua_5.0

bookmark · Added on March 18, 2025 · 33 min read

Bloom’s 3 Stages of Talent Development

bookmark · Added on March 18, 2025 · 1 min read

First, fun and exciting playtime. Then, intense and strenuous skill development. Finally, developing one’s individual style while pushing the boundaries of the field.

introduction-to-algorithms-and-machine-learning

bookmark · Added on March 18, 2025 · 3h 33m read

justinmath-linearAlgebra

bookmark · Added on March 18, 2025 · 1h 3m read

Russell’s Paradox and Possible Solutions

bookmark · Added on March 18, 2025 · 6 min read

The origins of set theory can be traced back to a Bohemian priest, Bernhard Bolzano (1781-1848), who was a professor of religion at the University of Prague.

The Making of Python

bookmark · Added on March 17, 2025 · 9 min read

Guido van Rossum is the author of Python, an interpreted, interactive object-oriented programming language.

Advice on Upskilling

bookmark · Added on March 17, 2025 · 1h 27m read

tt-metal/METALIUM_GUIDE.md at main · tenstorrent/tt-metal · GitHub

bookmark · Added on March 17, 2025 · 11 min read

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model. - tenstorrent/tt-metal

Scoping out the Tenstorrent Wormhole

bookmark · Added on March 17, 2025 · 1 min read

The Tenstorrent Wormhole n300s PCIe accelerator board is available for purchase, featuring 672 RISC-V cores driving 466 TFLOP/s of FP8 matmul.

What’s the (floating) Point of all these data types? A (not so) brief overview of the history and usage of datatypes within the wide world of computation

bookmark · Added on March 17, 2025 · 1 min read

This presentation delves into the fascinating and sometimes aggravating world of numerical data types, exploring the evolution, strengths, and weaknesses of decimal, fixed point, floating point, and shared exponent formats over the past 70 years.

Physics of language models

bookmark · Added on March 17, 2025 · 4 min read

Many asked about collaborations (details are in FAQ). Short answer: unless you're from Meta and willing to work with us in your spare time (20+ hrs/week), or you're an early-year PhD from UCB/NYU/CMU/UW (but application ddl was Jan 10, 2025). Citation request: I'm delighted to know that multiple

Tenstorrent first thoughts

bookmark · Added on March 17, 2025 · 4 min read

I've looked into alternative AI accelerators to continue my saga of running GGML on lower power-consumption hardware. The most promising - and the only one that ever replied to my emails - was Tenstorrent. This post is me deeply thinking about if buying their hardware for development is a good inve ...

7 x 11.5 long title.p65

bookmark · Added on March 10, 2025 · 11 min read

Neural Networks, Manifolds, and Topology

bookmark · Added on March 9, 2025 · 15 min read

However, there remain a number of concerns about them. One is that it can be quite challenging to understand what a neural network is really doing.

Attention from Beginners Point of View

bookmark · Added on March 9, 2025 · 2 min read

Transformers are a type of neural network architecture which is popularly used for text generations, machine translations, etc.

(How) Do Language Models Track State?

bookmark · Added on March 9, 2025 · 1 min read

Transformer language models (LMs) exhibit behaviors -- from storytelling to code generation -- that appear to require tracking the unobserved state of an evolving world. How do they do so? We study state tracking in LMs trained or fine-tuned to compose permutations (i.e., to compute the order of a set of objects after a sequence of swaps). Despite the simple algebraic structure of this problem, many other tasks (e.g., simulation of finite automata and evaluation of boolean expressions) can be reduced to permutation composition, making it a natural model for state tracking in general. We show that LMs consistently learn one of two state tracking mechanisms for this task. The first closely resembles the "associative scan" construction used in recent theoretical work by Liu et al. (2023) and Merrill et al. (2024). The second uses an easy-to-compute feature (permutation parity) to partially prune the space of outputs, then refines this with an associative scan. The two mechanisms exhibit markedly different robustness properties, and we show how to steer LMs toward one or the other with intermediate training tasks that encourage or suppress the heuristics. Our results demonstrate that transformer LMs, whether pretrained or fine-tuned, can learn to implement efficient and interpretable state tracking mechanisms, and the emergence of these mechanisms can be predicted and controlled.

Why Attention Is All You NeedWhy Attention Is All You Need

bookmark · Added on March 9, 2025 · 10 min read

The Transformer architecture introduced in this paper was a major breakthrough in sequence transduction methodologies, particularly within neural machine translation (NMT) and broader natural language processing (NLP).

CFD Python: 12 steps to Navier-Stokes

bookmark · Added on March 7, 2025 · 4 min read

We announce the public release of online educational materials for self-learners of CFD using IPython Notebooks: the CFD Python Class!

tt-mlir documentation

bookmark · Added on March 6, 2025 · 11 min read

The following document provides an overview of the TT-MLIR project, with a focus on the technical specifications of an MLIR-based compiler stack. So what exactly is an MLIR-based compiler stack?

Tutorials

bookmark · Added on March 6, 2025 · 1 min read

Multi-Level IR Compiler Framework

Yizhou Shan's Home Page

bookmark · Added on March 6, 2025 · 11 min read

This paper has a really nice Intro, pay close attention to how they lay out the storyline.

Attention in SRAM on Tenstorrent Grayskull

bookmark · Added on March 3, 2025 · 29 min read

When implementations of the Transformer's self-attention layer utilize SRAM instead of DRAM, they can achieve significant speedups. The Tenstorrent Grayskull architecture provides a large SRAM, distributed across a grid of cores. This work presents a fused kernel for Grayskull, that exclusively utilizes its large SRAM by combining matrix multiplication, attention score scaling and Softmax operations. Additionally, a dedicated Softmax kernel utilizing the SRAM and a CPU implementation serving as a baseline are presented. The Softmax operation consumes most of the runtime in the computation of attention weights from queries and keys on Grayskull. The speedup of the dedicated Softmax kernel compared to the CPU implementation is up to $10 \times$, and the Softmax implementation inside the fused kernel is approximately $1.8 \times$ faster than the dedicated Softmax kernel. The time and memory complexity of all implementations is quadratic in sequence length. Currently, the Grayskull e150 is approximately $30 \times$ cheaper for the general public than an Nvidia H100 PCIe (a state-of-the-art GPU) and offers approximately $1.5 \times$ more SRAM.

Crossing the uncanny valley ofconversational voice

bookmark · Added on March 1, 2025 · 11 min read

At Sesame, our goal is to achieve “voice presence”—the magical quality that makes spoken interactions feel real, understood, and valued.

Project 3: Distributed File System

bookmark · Added on February 28, 2025 · 19 min read

By John Taylor Gatto

bookmark · Added on February 27, 2025 · 1 min read

4) The differentiating function.

How to Think About TPUs

bookmark · Added on February 26, 2025 · 25 min read

All about how TPUs work, how they're networked together to enable multi-chip training and inference, and how they limit the performance of our favorite algorithms. While this may seem a little dry, it's super important for actually making models efficient.

Programming Really Is Simple Mathematics

bookmark · Added on February 25, 2025 · 1 min read

A re-construction of the fundamentals of programming as a small mathematical theory (PRISM) based on elementary set theory. Highlights: $\bullet$ Zero axioms. No properties are assumed, all are proved (from standard set theory). $\bullet$ A single concept covers specifications and programs. $\bullet$ Its definition only involves one relation and one set. $\bullet$ Everything proceeds from three operations: choice, composition and restriction. $\bullet$ These techniques suffice to derive the axioms of classic papers on the "laws of programming" as consequences and prove them mechanically. $\bullet$ The ordinary subset operator suffices to define both the notion of program correctness and the concepts of specialization and refinement. $\bullet$ From this basis, the theory deduces dozens of theorems characterizing important properties of programs and programming. $\bullet$ All these theorems have been mechanically verified (using Isabelle/HOL); the proofs are available in a public repository. This paper is a considerable extension and rewrite of an earlier contribution [arXiv:1507.00723]

Tenstorrent Wormhole Series Part 1: Physicalities

bookmark · Added on February 25, 2025 · 5 min read

A company called Tenstorrent design and sell PCIe cards for AI acceleration. At the time of writing, they've recently started shipping their Wormhole n150s and Wormhole n300s cards.

Community Highlight: Tenstorrent Wormhole Series Part 2: Which disabled rows?

bookmark · Added on February 25, 2025 · 5 min read

An in depth look at Tenstorrent Wormhole, originally posted on corsix.org

Execution-based Code Generation using Deep Reinforcement Learning

bookmark · Added on February 20, 2025 · 1h 1m read

The utilization of programming language (PL) models, pre-trained on large-scale code corpora, as a means of automating software engineering processes has demonstrated considerable potential in streamlining various code generation tasks such as code completion, code translation, and program synthesis. However, current approaches mainly rely on supervised fine-tuning objectives borrowed from text generation, neglecting unique sequence-level characteristics of code, including but not limited to compilability as well as syntactic and functional correctness. To address this limitation, we propose PPOCoder, a new framework for code generation that synergistically combines pre-trained PL models with Proximal Policy Optimization (PPO) which is a widely used deep reinforcement learning technique. By utilizing non-differentiable feedback from code execution and structure alignment, PPOCoder seamlessly integrates external code-specific knowledge into the model optimization process. It's important to note that PPOCoder is a task-agnostic and model-agnostic framework that can be used across different code generation tasks and PLs. Extensive experiments on three code generation tasks demonstrate the effectiveness of our proposed approach compared to SOTA methods, achieving significant improvements in compilation success rates and functional correctness across different PLs.

þÿThe Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers

bookmark · Added on February 19, 2025 · 1h 44m read

neural video codecs: the future of video compression

bookmark · Added on February 17, 2025 · 23 min read

how deep learning could rewrite the way we encode and decode video

Lego Mindset vs. Woodworking Mindset

bookmark · Added on February 17, 2025 · 10 min read

On Accepting Toil

Unnamed Document

bookmark · Added on February 17, 2025 · 1 min read

Mastering LLM Techniques: Evaluation

bookmark · Added on February 15, 2025 · 10 min read

Evaluating large language models (LLMs) and retrieval-augmented generation (RAG) systems is a complex and nuanced process, reflecting the sophisticated and multifaceted nature of these systems.

Mastering LLM Inference Techniques: Inference Optimization

bookmark · Added on February 15, 2025 · 1 min read

Learn about the most pressing challenges in LLM inference, along with some practical solutions.

Automating GPU Kernel Generation with DeepSeek-R1 and Inference Time Scaling

bookmark · Added on February 15, 2025 · 3 min read

As AI models extend their capabilities to solve more sophisticated challenges, a new scaling law known as test-time scaling or inference-time scaling is emerging. Also known as AI reasoning or long…

The high-return activity of raising others’ aspirations

bookmark · Added on February 12, 2025 · 1 min read

Yesterday I had lunch with a former Ph.D student of mine, who is now highly successful and tenured at a very good school. I was reminded that, over twenty years ago, I was Graduate Director of Admissions. One of my favorite strategies was to take strong candidates who applied for Masters and also offer them […]

Build Your Own Text Editor

bookmark · Added on January 27, 2025 · 1 min read

The text editor is antirez’s kilo, with some changes.

The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey

bookmark · Added on January 27, 2025 · 2h 19m read

The advent of Large Language Models (LLMs) represents a notable breakthrough in Natural Language Processing (NLP), contributing to substantial progress in both text comprehension and generation. However, amidst these advancements, it is noteworthy that LLMs often face a limitation in terms of context length extrapolation. Understanding and extending the context length for LLMs is crucial in enhancing their performance across various NLP applications. In this survey paper, we delve into the multifaceted aspects of exploring why it is essential, and the potential transformations that superior techniques could bring to NLP applications. We study the inherent challenges associated with extending context length and present an organized overview of the existing strategies employed by researchers. Additionally, we discuss the intricacies of evaluating context extension techniques and highlight the open challenges that researchers face in this domain. Furthermore, we explore whether there is a consensus within the research community regarding evaluation standards and identify areas where further agreement is needed. This comprehensive survey aims to serve as a valuable resource for researchers, guiding them through the nuances of context length extension techniques and fostering discussions on future advancements in this evolving field.

Tilde, my LLVM alternative

bookmark · Added on January 25, 2025 · 2 min read

I'm Yasser and I've made it my mission to produce an alternative to LLVM, the current king of compiler backend libraries.

A WebAssembly compiler that fits in a tweet

bookmark · Added on January 25, 2025 · 16 min read

Starting with a 192-byte one-liner that implements a Reverse Polish Notation arithmetic compiler, we'll work backward to transform it into readable JavaScript by removing one code golf trick at a time

Transformer Memory as a Differentiable Search Index

bookmark · Added on January 25, 2025 · 35 min read

In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot setup.

history.dvi

bookmark · Added on January 25, 2025 · 13 min read

Proof of correctness of data representation

bookmark · Added on January 25, 2025 · 1 min read

Elements of Programming

bookmark · Added on January 23, 2025 · 4h 59m read

Unveiling_DeepSeek.pdf

bookmark · Added on January 22, 2025 · 8 min read

successful modifications since its inception, let alone large-scale validation.

Stating the problem in Lean

bookmark · Added on January 19, 2025 · 11 min read

Note: this post was written for Lean 3; the latest version, Lean 4, is a very different language. Turn back the clock to 2009: a confused physics major newly infatuated with math and computer science, I enrolled in MATH 273: Numbers and Proofs at the University of Calgary. This wasn’t my first encounter with mathematical proof; in first-year calculus I’d mastered rote regurgitation of delta-epsilon proofs. Despite writing out several dozen, their meaning never progressed beyond a sort of incantation I can summon to this day (for every $ \epsilon > 0 $ there exists a $ \delta > 0 $ such that…).

DeepSeek-V3 Explained: A Deep Dive into the Next-Generation AI Model

bookmark · Added on January 18, 2025 · 9 min read

Artificial Intelligence (AI) is advancing at an unprecedented pace, and the DeepSeek-V3 model is at the forefront of this revolution. As…

Foundations of Large Language Models

bookmark · Added on January 17, 2025 · 1 min read

This is a book about large language models. As indicated by the title, it primarily focuses on foundational concepts rather than comprehensive coverage of all cutting-edge technologies. The book is structured into four main chapters, each exploring a key area: pre-training, generative models, prompting techniques, and alignment methods. It is intended for college students, professionals, and practitioners in natural language processing and related fields, and can serve as a reference for anyone interested in large language models.

þÿA comprehensive study of Convergent and Commutative Replicated Data Types

bookmark · Added on January 15, 2025 · 1h 17m read

hist

bookmark · Added on January 12, 2025 · 35 min read

þÿPart II - Algebraic Topology (Theorems with proof)

bookmark · Added on January 11, 2025 · 53 min read

fundamental-group

bookmark · Added on January 11, 2025 · 1h 8m read

Category Theory: Lecture Notes and Online Books

bookmark · Added on January 10, 2025 · 8 min read

The links below are to various freely (and legitimately!) available online mathematical resources for those interested in category theory at an elementary/intermediate level. There is supplementary page, introductory readings for philosophers, for reading suggestions for those looking for the most accessible routes into category theory and/or links to philosophical discussions. A gentle introduction? My Category … Category Theory: Lecture Notes and Online Books Read More »

Why Futhark?

bookmark · Added on January 9, 2025 · 1 min read

A high-performance and high-level purely functional data-parallel array programming language that can execute on the GPU and CPU.

DeepSeek-V3 Technical Report

bookmark · Added on January 7, 2025 · 1h 51m read

We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3.

gcvsmalloc

bookmark · Added on January 2, 2025 · 52 min read

—————SOURCES———————————————————————— Percolation – Béla Bollobás and Oliver Riordan Cambridge University Press, New York, 2006. Sixty Years of Percolation – Hugo Duminil-Copin https://www.ihes.fr/~duminil/publi/2018ICM.pdf Percolation – Geoffrey Grimmett volume 321 of Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Springer-Verlag, Berlin, second edition, 1999. —————NOTES————————————————————————— Note at 10:42 – The uniqueness of the infinite cluster is known for the d-dimenional lattice since the works of Aizenman, Kesten and Newman - [Uniqueness of the infinite cluster and continuity of connectivity functions for short and long range percolation (1987)] and Burton and Keane - [Density and uniqueness in percolation (1989)]. It does not hold in general: when the graph in question is a regular tree for example, there are always infinitely many clusters during the supercritical phase. The two last results shown here are only known for site percolation (in which vertices are open or closed instead of edges) in the triangular lattice, where a scaling limit for the boundaries of critical clusters was proved to exist (more on that in the third note). It is believed that these results are universal, that is, valid in great generality for planar percolation processes near criticality. The third result is from an appendix by Gábor Pete in the paper [Scaling limits for the threshold window: When does a monotone Boolean function flip its outcome? (2017)] by Ahlberg and Steif. Consider an n by n box, and the event where there exists a left-right crossing of said box. Recall the uniform coupling from the video: intuitively, the result is saying that the point at which this crossing emerges in the uniform coupling is with high probability inside an interval of size n^{-3/4} around 1/2. The fourth result is saying that the average size of the cluster of the origin (or any other given point) goes to infinity as we let p approach the critical parameter like a specific power of the distance between p and p_c. This power is called a critical exponent. The existence of these exponents was proved by Smirnov and Werner in the paper [Critical exponents for two-dimensional percolation (2001)]. Note at 10:52 – Hugo Duminil-Copin has several major contributions to the study of processes arising in statistical physics, including Bernoulli percolation. Among his works on Ising and Ising-like processes we can cite [Random Currents and Continuity of Ising Model’s Spontaneous Magnetization (2015)] with Aizenman and Sidoravicius and [Sharp phase transition for the random-cluster and Potts models via decision trees (2019)] with Raoufi and Tassion. Note at 12:38 – In the triangular lattice site percolation, Stanislav Smirnov proved the conformal invariance of crossing probabilities at criticality (see https://www.unige.ch/~smirnov/papers/icmp-final.pdf for an overview), which led to the proof of the existence of scaling limits of exploration curves as Schramm–Loewner evolution processes. See [Critical percolation in the plane (2009)] by Smirnov. This provided a deep understanding of the critical phase in the triangular lattice site percolation, which to this day is not extended to the square lattice. Note at 17:52 – It is not at all obvious that the probability of being connected to infinity is continuous above criticality. This result can be proved in the d-dimenional hypercubic lattices using the uniqueness of the infinite cluster, and more generally it was proved for transitive graphs (intuitively, graphs in which all vertices look the same) by Häggström, Peres and Schonmann in [Percolation on transitive graphs as a coalescent process: Relentless merging followed by simultaneous uniqueness (1999)]. —————SECTIONS——————————————————————— 0:00 Introduction 1:37 Definition – Bernoulli Percolation 5:23 Definition – Uniform Coupling 7:56 Exploration – High-Resolution Square Grid 9:40 Exploration – Questions and Kesten's Theorem 10:58 Exploration – Ising Model 11:54 Exploration – Critical Percolation 12:50 Exploration – Three-Dimensional Cubic Lattice and Beyond 14:13 Proof – Theorem Statement 15:14 Proof – Simplifications 16:29 Proof – Definition of Critical Parameter 18:41 Proof – Critical Parameter is Greater Than Zero 20:44 Proof – Duality Definition 21:56 Proof – Critical Parameter is Less Than One 25:16 Proof – Summary and Idea for Kesten's Theorem 26:11 Conclusion —————CREDITS———————————————————————— Caio Alves – writing, 3D animation Aranka Hrušková – writing, clarinet Vilas Winstein – writing, 2D animation, editing, voice-over Special thanks to Anisah Awad, Gábor Pete, Jyotsna Sreenivasan, Angie Zavala This video is an entry in the second Summer of Mathematics Exposition (#SoME2) The photographs used in this video are licensed under the Creative Commons Attribution-ShareAlike license: https://creativecommons.org/licenses/by-sa/4.0/deed.en Uploader: Spectral Collective Duration: 1612s Views: 455517

Bloom filters debunked: Dispelling 30 Years of bad math with Coq!

bookmark · Added on December 27, 2024 · 11 min read

While conceptually simple, this feature actually requires more engineering effort than one would expect - in particular, tracking the set of known malicious URLs in a practical manner turns out to be somewhat difficult.

DeepSeek-V3/DeepSeek_V3.pdf at main · deepseek-ai/DeepSeek-V3

bookmark · Added on December 26, 2024 · 1 min read

by Marcus Hutter and David Quarel and Elliot Catt

bookmark · Added on December 24, 2024 · 4 min read

The book can be ordered from amazon. com / co.

Deepseek: The Quiet Giant Leading China’s AI Race

bookmark · Added on December 24, 2024 · 25 min read

Annotated translation of its CEO's deepest interview

The Double-E Infix Expression Parsing Method

bookmark · Added on December 23, 2024 · 6 min read

Topic in Programming Models

Demystifying Debuggers, Part 2: The Anatomy Of A Running Program

bookmark · Added on December 23, 2024 · 29 min read

On the concepts involved in a running program. What happens, exactly, when you double click an executable file, or launch it from the command line, and it begins to execute?

Towards a Categorical Foundation of Deep Learning: A Survey

bookmark · Added on December 22, 2024 · 1 min read

The unprecedented pace of machine learning research has lead to incredible advances, but also poses hard challenges. At present, the field lacks strong theoretical underpinnings, and many important achievements stem from ad hoc design choices which are hard to justify in principle and whose effectiveness often goes unexplained. Research debt is increasing and many papers are found not to be reproducible. This thesis is a survey that covers some recent work attempting to study machine learning categorically. Category theory is a branch of abstract mathematics that has found successful applications in many fields, both inside and outside mathematics. Acting as a lingua franca of mathematics and science, category theory might be able to give a unifying structure to the field of machine learning. This could solve some of the aforementioned problems. In this work, we mainly focus on the application of category theory to deep learning. Namely, we discuss the use of categorical optics to model gradient-based learning, the use of categorical algebras and integral transforms to link classical computer science to neural networks, the use of functors to link different layers of abstraction and preserve structure, and, finally, the use of string diagrams to provide detailed representations of neural network architectures.

Soft question: Deep learning and higher categories

bookmark · Added on December 22, 2024 · 1 min read

Recently, I have stumbled upon certain articles and lecture videos that use category theory to explain certain aspects of machine learning or deep learning (e.g. Cats for AI and the paper An enriched

Algebraic Databases

bookmark · Added on December 22, 2024 · 1 min read

Databases have been studied category-theoretically for decades. The database schema---whose purpose is to arrange high-level conceptual entities---is generally modeled as a category or sketch. The data itself, often called an instance, is generally modeled as a set-valued functor, assigning to each conceptual entity a set of examples. While mathematically elegant, these categorical models have typically struggled with representing concrete data such as integers or strings. In the present work, we propose an extension of the set-valued functor model, making use of multisorted algebraic theories (a.k.a. Lawvere theories) to incorporate concrete data in a principled way. This also allows constraints and queries to make use of operations on data, such as multiplication or comparison of numbers, helping to bridge the gap between traditional databases and programming languages. We also show how all of the components of our model---including schemas, instances, change-of-schema functors, and queries - fit into a single double categorical structure called a proarrow equipment (a.k.a. framed bicategory).

Categorical Databases

bookmark · Added on December 22, 2024 · 10 min read

walter

bookmark · Added on December 22, 2024 · 25 min read

FPGAs for Software Engineers 0: The Basics

bookmark · Added on December 22, 2024 · 22 min read

A brief introduction to FPGAs, Verilog and simulation

Data-Oriented Design

bookmark · Added on December 19, 2024 · 38 min read

Data-Oriented Design

A note about "The Humane Representation of Thought"

bookmark · Added on December 17, 2024 · 1 min read

A year and a half ago, on a plane, I wrote An Ill-Advised Personal Note about "Media for Thinking the Unthinkable".

BLT__Patches_Scale_Better_Than_Tokens

bookmark · Added on December 17, 2024 · 1h 3m read

On Ousterhout’s Dichotomy Oct 6, 2024

bookmark · Added on December 17, 2024 · 3 min read

Why are there so many programming languages? One of the driving reasons for this is that some languages tend to produce fast code, but are a bit of a pain to use (C++), while others are a breeze to write, but run somewhat slow (Python). Depending on the ratio of CPUs to programmers, one or the other might be relatively more important.

The categorical abstract machine

bookmark · Added on December 17, 2024 · 3 min read

The Cartesian closed categories have been shown by several authors to provide the right framework of the model theory of λ-calculus. The second author…

Position: Categorical Deep Learning is an Algebraic Theory of All Architectures

bookmark · Added on December 17, 2024 · 1h 26m read

We present our position on the elusive quest for a general-purpose framework for specifying and studying deep learning architectures. Our opinion is that the key attempts made so far lack a coherent bridge between specifying constraints which models must satisfy and specifying their implementations. Focusing on building a such a bridge, we propose to apply category theory -- precisely, the universal algebra of monads valued in a 2-category of parametric maps -- as a single theory elegantly subsuming both of these flavours of neural network design. To defend our position, we show how this theory recovers constraints induced by geometric deep learning, as well as implementations of many architectures drawn from the diverse landscape of neural networks, such as RNNs. We also illustrate how the theory naturally encodes many standard constructs in computer science and automata theory.

Fundamental Components of Deep Learning: A category-theoretic approach

bookmark · Added on December 17, 2024 · 6h 12m read

Deep learning, despite its remarkable achievements, is still a young field. Like the early stages of many scientific disciplines, it is marked by the discovery of new phenomena, ad-hoc design decisions, and the lack of a uniform and compositional mathematical foundation. From the intricacies of the implementation of backpropagation, through a growing zoo of neural network architectures, to the new and poorly understood phenomena such as double descent, scaling laws or in-context learning, there are few unifying principles in deep learning. This thesis develops a novel mathematical foundation for deep learning based on the language of category theory. We develop a new framework that is a) end-to-end, b) unform, and c) not merely descriptive, but prescriptive, meaning it is amenable to direct implementation in programming languages with sufficient features. We also systematise many existing approaches, placing many existing constructions and concepts from the literature under the same umbrella. In Part I we identify and model two main properties of deep learning systems parametricity and bidirectionality by we expand on the previously defined construction of actegories and Para to study the former, and define weighted optics to study the latter. Combining them yields parametric weighted optics, a categorical model of artificial neural networks, and more. Part II justifies the abstractions from Part I, applying them to model backpropagation, architectures, and supervised learning. We provide a lens-theoretic axiomatisation of differentiation, covering not just smooth spaces, but discrete settings of boolean circuits as well. We survey existing, and develop new categorical models of neural network architectures. We formalise the notion of optimisers and lastly, combine all the existing concepts together, providing a uniform and compositional framework for supervised learning.

Logic and linear algebra: an introduction

bookmark · Added on December 17, 2024 · 1 min read

We give an introduction to logic tailored for algebraists, explaining how proofs in linear logic can be viewed as algorithms for constructing morphisms in symmetric closed monoidal categories with additional structure. This is made explicit by showing how to represent proofs in linear logic as linear maps between vector spaces. The interesting part of this vector space semantics is based on the cofree cocommutative coalgebra of Sweedler.

Gemini: A Family of Highly Capable Multimodal Models

bookmark · Added on December 17, 2024 · 2h 29m read

This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models responsibly to users through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud Vertex AI.

Flow Matching Guide and Code

bookmark · Added on December 17, 2024 · 2h 56m read

Flow Matching (FM) is a recent framework for generative modeling that has achieved state-of-the-art performance across various domains, including image, video, audio, speech, and biological structures. This guide offers a comprehensive and self-contained review of FM, covering its mathematical foundations, design choices, and extensions. By also providing a PyTorch package featuring relevant examples (e.g., image and text generation), this work aims to serve as a resource for both novice and experienced researchers interested in understanding, applying and further developing FM.

Logical Complexity of Proofs

bookmark · Added on December 17, 2024 · 3 min read

If you cannot find proofs, talk about them. Robert Reckhow with his advsior Stephen Cook famously started the formal study of the complexity of proofs with their 1979 paper. They were interested in…

Mastering Board Games by External and Internal Planning with Language Models

bookmark · Added on December 17, 2024 · 2h 43m read

Proofs and Types

bookmark · Added on December 16, 2024 · 4h 10m read

Richard Hamming - Wikipedia

bookmark · Added on December 16, 2024 · 16 min read

Richard Wesley Hamming (February 11, 1915 – January 7, 1998) was an American mathematician whose work had many implications for computer engineering and telecommunications.

What is the "question" that programming language theory is trying to answer?

bookmark · Added on December 16, 2024 · 2 min read

I've been interested in various topics like Combinatory Logic, Lambda Calculus, Functional Programming for a while and have been studying them. However, unlike the "Theory of Computation" which str...

Introducing Limbo: A complete rewrite of SQLite in Rust

bookmark · Added on December 16, 2024 · 7 min read

we forked SQLite with the libSQL project. What would it be like if we just rewrote it?

TLA+ is hard to learn

bookmark · Added on December 16, 2024 · 5 min read

I’m a fan of the formal specification language TLA+. With TLA+, you can build models of programs or systems, which helps to reason about their behavior. TLA+ is particularly useful for reason…

How hard is constraint programming?

bookmark · Added on December 16, 2024 · 2 min read

Writing code using the Z3 SMT solver is different from typical programming, due to mixed programming models--not unlike CUDA for GPUs. Here's what to expect.

Geeks, MOPs, and sociopaths in subculture evolution

bookmark · Added on December 16, 2024 · 11 min read

How muggles and sociopaths invade and undermine creative subcultures; and how to stop them.

Advanced programming languages

bookmark · Added on December 16, 2024 · 13 min read

Students often ask for a recommendation on what language they should learn next.

ugh.book

bookmark · Added on December 16, 2024 · 7h 31m read

Working memory - Wikipedia

bookmark · Added on December 16, 2024 · 1h 6m read

Working memory is a cognitive system with a limited capacity that can hold information temporarily. [1] It is important for reasoning and the guidance of decision-making and behavior.

Working hurts less than procrastinating, we fear the twinge of starting

bookmark · Added on December 16, 2024 · 4 min read

When you procrastinate, you're probably not procrastinating because of the pain of working. …

llama.cpp guide - Running LLMs locally, on any hardware, from scratch

bookmark · Added on December 16, 2024 · 1h 9m read

Psst, kid, want some cheap and small LLMs?

GitHub - avinassh/py-caskdb: (educational) build your own disk based KV store

bookmark · Added on December 16, 2024 · 5 min read

(educational) build your own disk based KV store. Contribute to avinassh/py-caskdb development by creating an account on GitHub.

Depth-First Procrastination

bookmark · Added on December 15, 2024 · 1 min read

When subgoals recur infinitely.

Command Line Interface Guidelines

bookmark · Added on December 13, 2024 · 52 min read

An open-source guide to help you write better command-line programs, taking traditional UNIX principles and updating them for the modern day.

How Many Computers Are In Your Computer?

bookmark · Added on December 11, 2024 · 13 min read

Any ‘computer’ is made up of hundreds of separate computers plugged together, any of which can be hacked. I list some of these parts.

Category theory for scientists (Old version)

bookmark · Added on December 11, 2024 · 8h 36m read

There are many books designed to introduce category theory to either a mathematical audience or a computer science audience. In this book, our audience is the broader scientific community. We attempt to show that category theory can be applied throughout the sciences as a framework for modeling phenomena and communicating results. In order to target the scientific audience, this book is example-based rather than proof-based. For example, monoids are framed in terms of agents acting on objects, sheaves are introduced with primary examples coming from geography, and colored operads are discussed in terms of their ability to model self-similarity. A new version with solutions to exercises will be available through MIT Press.

Genie 2: A large-scale foundation world model

bookmark · Added on December 10, 2024 · 7 min read

Generating unlimited diverse training environments for future general agents

Design Of This Website

bookmark · Added on December 9, 2024 · 1h 37m read

Meta page describing Gwern.net, the self-documenting website’s implementation and experiments for better ‘semantic zoom’ of hypertext; technical decisions using Markdown and static hosting.

WilliamYi96/Awesome-Energy-Based-Models: A curated list of resources on energy-based models.

bookmark · Added on December 9, 2024 · 7 min read

A curated list of resources on energy-based models. - WilliamYi96/Awesome-Energy-Based-Models

"CBLL, Research Projects, Computational and Biological Learning Lab, Courant Institute, NYU"

bookmark · Added on December 9, 2024 · 6 min read

Yann LeCun's Web pages at NYU

yataobian/awesome-ebm: Collecting research materials on EBM/EBL (Energy Based Models, Energy Based Learning)

bookmark · Added on December 9, 2024 · 16 min read

Collecting research materials on EBM/EBL (Energy Based Models, Energy Based Learning) - yataobian/awesome-ebm

TuringConf

bookmark · Added on December 7, 2024 · 25 min read

Omens of exceptional talent

bookmark · Added on December 6, 2024 · 3 min read

Gaiseric…was a man of moderate height and lame in consequence of a fall from his horse. He was a man of deep thought and few words I’m often asked about the signs of exceptional talent I’ve observed, probably because I spend too much running around talking to people & observing things, instead of doing anything useful. Patrick Collison, Sam Altman, and Tyler Cowen are the three names that come to mind when thinking about this question. Of my writing, Intelligence killed …

An Introduction to Current Theories of Consciousness

bookmark · Added on December 6, 2024 · 1h 11m read

(Crosspost from my blog) • • There are few academic lists of theories of consciousness (Doerig 2020) as well as some good blog post series about specific ideas (shout out to SelfAwarePatterns), but…

Being the (Pareto) Best in the World

bookmark · Added on December 6, 2024 · 4 min read

John Wentworth argues that becoming one of the best in the world at *one* specific skill is hard, but it's not as hard to become the best in the worl…

Greg Yang

bookmark · Added on December 5, 2024 · 7 min read

I am currently developing a framework called Tensor Programs for understanding large neural networks.

A Century of Mathematics in America, Part I

bookmark · Added on December 4, 2024 · 1h 35m read

Fastest contributed programs, grouped by programming language implementation

bookmark · Added on December 3, 2024 · 1 min read

Charts showing benchmark program performance grouped by implementation language.

Haskell as fast as C: working at a high altitude for low level performance

bookmark · Added on December 3, 2024 · 15 min read

After the last post about high performance, high level programming, Slava Pestov, of Factor fame, wondered whether it was generally true that “if you want good performance you have to write C…

On Competing with C Using Haskell

bookmark · Added on December 3, 2024 · 13 min read

Mark Karpov wrote in his article on Migrating text metrics to pure Haskell how he originally did foreign calls out to C for many of the functions in his text metric package, but now ported them to Haskell when he learned that Haskell can give you performance comparable to C.

Performance

bookmark · Added on December 3, 2024 · 6 min read

Moreover, it's often not clear if two programs which supposedly have the same functionality really do the same thing.

TS_Tutorial

bookmark · Added on December 3, 2024 · 2h 13m read

Category Theory usage in Algebraic Topology

bookmark · Added on December 3, 2024 · 1 min read

First my question: How much category theory should someone studying algebraic topology generally know? Motivation: I am taking my first graduate course in algebraic topology next semester, and,...

Topos Theory in a Nutshell

bookmark · Added on December 3, 2024 · 12 min read

Okay, you wanna know what a topos is? First I'll give you a hand-wavy vague explanation, then an actual definition, then a few consequences of this definition, and then some examples.

context

bookmark · Added on December 3, 2024 · 9h 23m read

Proof Explorer

bookmark · Added on December 3, 2024 · 2h 0m read

Inspired by Whitehead and Russell's monumental Principia Mathematica, the Metamath Proof Explorer has over 26,000 completely worked out proofs in its main sections (and over 41,000 counting "mathboxes", which are annexes where contributors can develop additional topics), starting from the very foundation that mathematics is built on and eventually arriving at familiar mathematical facts and beyond.

An Invitation to Applied Category Theory

bookmark · Added on December 3, 2024 · 1 min read

Abstract page for arXiv paper 1803.05316: Seven Sketches in Compositionality: An Invitation to Applied Category Theory

An Invitation to Applied Category Theory

bookmark · Added on December 3, 2024 · 2 min read

Cambridge Core - Programming Languages and Applied Logic - An Invitation to Applied Category Theory

Introducing io_uring_spawn

bookmark · Added on December 2, 2024 · 24 min read

The traditional mechanism for launching a program in a new process on Unix systems—forking and execing—has been with us for decades, but it is not really the most efficient of operations.

Fast Multidimensional Matrix Multiplication on CPU from Scratch

bookmark · Added on July 30, 2024 · 15 min read

Numpy can multiply two 1024x1024 matrices on a 4-core Intel CPU in ~8ms.This is incredibly fast, considering this boils down to 18 FLOPs / core / cycle, with...

Efficient n-states on x86 systems

bookmark · Added on July 29, 2024 · 4 min read

The text discusses how to efficiently handle control flow in x86 systems when a flag can have multiple states beyond true and false. It explains how to use condition codes, such as testing for zero and parity, to minimize the number of instructions needed for these tests. Additionally, it touches on the challenges and limitations of using inline assembly for optimization in C programming.

Program tuning as a resource allocation problem

bookmark · Added on July 29, 2024 · 6 min read

Program tuning involves balancing simplicity and performance while sharing cache resources among various subsystems. Optimizing one function can impact others, making it a global resource allocation problem that requires careful consideration of algorithms and their resource footprints. Better tools and metrics are needed to manage and analyze cache resource consumption effectively.

How web bloat impacts users with slow connections

bookmark · Added on July 29, 2024 · 59 min read

Web bloat makes many websites difficult to use for people with slow internet connections and devices. Sites like Discourse and Reddit perform poorly on low-end devices, even if they seem fast on high-end ones. Improving web performance for these users is crucial, as many people rely on older, slower devices.

Files are hard

bookmark · Added on July 29, 2024 · 24 min read

Writing files in a way that ensures their robustness is challenging due to the complexity involved. The paper discusses various issues related to file corruption and data loss, such as crash consistency, filesystem semantics, filesystem correctness, error handling, and error recovery. It highlights the differences in how different filesystems handle errors and points out bugs and inconsistencies found in popular filesystems. The paper also addresses the frequency of disk errors and data corruption, emphasizing the need for caution when writing files and the importance of using libraries or tools to ensure safety. Overall, the document emphasizes the difficulty of reasoning about file-related problems and the need for careful considerations when working with filesystems.

Ringing in a new asynchronous I/O API

bookmark · Added on July 29, 2024 · 9 min read

The new "io_uring" interface simplifies asynchronous I/O in the Linux kernel by using two ring buffers for submission and completion queues. Applications can set up these buffers with a system call and submit I/O requests through a structured format. This method aims to reduce complaints about AIO by improving efficiency and ease of use.

applicative-mental-models

bookmark · Added on July 29, 2024 · 6 min read

The text discusses the importance of understanding program performance for effective optimization. It emphasizes that while most optimizations may not be necessary, being aware of critical performance paths is essential. The author provides latency numbers to help programmers grasp the impact of different operations on performance.

Optimizing subroutines in assembly language

bookmark · Added on July 29, 2024 · 5h 19m read

Optimizing subroutines in assembly language involves various techniques such as using inline assembly in a C++ compiler, separating code using MMX registers from code using ST registers, and understanding different register sizes and memory operands. It is important to consider the use of instruction prefixes, intrinsic functions for vector operations, and accessing class and structure members efficiently. Additionally, preventing false dependences, aligning loop and subroutine entries, and optimizing instruction sizes can improve performance. However, it is crucial to note that these optimizations are processor-specific and may vary depending on the target platform.

Demystifying programs that create programs, part 1: A disassembler

bookmark · Added on July 29, 2024 · 16 min read

This blog post starts a series on creating programs that demystify how programs work. The first program is a disassembler that reads bytecode and converts it into assembly language, while a future post will cover creating an assembler. The disassembler uses a table of mnemonics and instruction sizes to print out the corresponding assembly instructions from bytecode.

QBE vs LLVM

bookmark · Added on July 29, 2024 · 2 min read

QBE and LLVM are both compiler backends, but QBE is a smaller, more accessible project aimed at amateur language designers. While LLVM is feature-rich and complex, QBE focuses on simplicity and efficiency, making it easier to use for quick projects. QBE provides straightforward operations and a cleaner intermediate language, reducing the complexity often found in LLVM.

Recent presentations and papers

bookmark · Added on July 29, 2024 · 9 min read

Andi Kleen's work focuses on improving Linux performance through various techniques like hardware monitoring and profiling. He has presented on topics such as lock elision, multi-core scalability, and error handling in the Linux kernel. His contributions include discussions on modern CPU performance, tools for Linux development, and enhancements for energy efficiency.

brotli-2015-09-22

bookmark · Added on July 29, 2024 · 6 min read

How long does it take to make a context switch?

bookmark · Added on July 29, 2024 · 14 min read

Context switching times vary significantly across different Intel CPU models, with more expensive CPUs generally performing better. The performance can be greatly affected by cache usage and thread migration between cores, leading to increased costs when tasks are switched. Optimizing the number of threads to match the number of hardware threads can improve CPU efficiency and reduce context switching overhead.

Ghostty Devlog 001

bookmark · Added on July 29, 2024 · 7 min read

Ghostty is a terminal emulator developed as a side project. In this devlog, the author shares details about the tech stack behind Ghostty, including its cross-platform capabilities and GPU acceleration. The devlog also introduces two features: automatic shell integration injection and auto-italicize fonts. The shell integration feature improves prompt redrawing, working directory reporting, and active process detection, while the auto-italicize fonts feature fixes a bug and adds the ability to skew regular fonts to create fake italics. The devlog concludes by inviting readers to follow the author on social media for updates and future devlogs.

Tiled Matrix Multiplication

bookmark · Added on July 29, 2024 · 5 min read

Tiled matrix multiplication is an efficient algorithm used on GPUs that reduces memory access by utilizing shared memory. By organizing threads into blocks, each thread can perform calculations more quickly and with fewer memory accesses. This method is important for improving performance in tasks like graphics rendering and machine learning.

Rust Atomics and Locks

bookmark · Added on July 29, 2024 · 3 min read

This book by Mara Bos explores Rust programming language's concurrency features, including atomics, locks, and memory ordering. Readers will gain a practical understanding of low-level concurrency in Rust, covering topics like mutexes and condition variables. The book provides insights on implementing correct concurrency code and building custom locking and synchronization mechanisms.

Compiler Backend

bookmark · Added on July 29, 2024 · 3 min read

The QBE compiler backend is designed to be a compact yet high-performance C embeddable backend that prioritizes correctness, simplicity, and user-friendliness. It compiles on various x64 operating systems and boasts features like IEEE floating point support, SSA-based intermediate language, and quick compilation times. While currently limited to x64 platforms, plans include ARM support and further enhancements. The backend has been successfully utilized in various projects, showcasing its adaptability and effectiveness in compiler development.

Vale's Memory Safety Strategy: Generational References and Regions

bookmark · Added on July 29, 2024 · 19 min read

Vale's memory safety strategy uses generational references to manage memory without relying on traditional methods like garbage collection. Each reference stores a "generation" ID, and before accessing an object, a check ensures the ID matches the object's current generation. This approach allows for efficient memory management while maintaining safety, reducing overhead significantly compared to other methods.

Introduction

bookmark · Added on July 29, 2024 · 4 min read

Wait-freedom ensures that each thread can progress independently, executing operations in a fixed number of steps without being blocked by others. Lock-freedom allows the system to make overall progress, but individual threads might still get stuck. Obstruction-freedom means a thread can only progress without interference from others, making it a weaker guarantee than lock-freedom.

Cache-Oblivious Algorithms

bookmark · Added on July 29, 2024 · 4 min read

Cache-oblivious algorithms are designed to use processor caches efficiently without needing to know specific cache details. They work by dividing data into smaller parts, allowing more computations to happen in cache and reducing memory access. This leads to better performance, especially in parallel algorithms, by minimizing shared memory bottlenecks.

A Memory Allocator

bookmark · Added on July 29, 2024 · 13 min read

A memory allocator is software that manages dynamic memory allocation in programs, providing functions like malloc(), free(), and realloc(). This particular allocator aims to minimize memory wastage and improve efficiency, and it is widely used in various systems, including Linux. It employs techniques like coalescing freed chunks and supports memory mapping to enhance performance and reduce fragmentation.

Cramming: Training a Language Model on a Single GPU in One Day

bookmark · Added on July 29, 2024 · 1h 2m read

Recent trends in language modeling have focused on increasing performance through scaling, and have resulted in an environment where training language models is out of reach for most researchers and practitioners. While most in the community are asking how to push the limits of extreme computation, we ask the opposite question: How far can we get with a single GPU in just one day? We investigate the downstream performance achievable with a transformer-based language model trained completely from scratch with masked language modeling for a single day on a single consumer GPU. Aside from re-analyzing nearly all components of the pretraining pipeline for this scenario and providing a modified pipeline with performance close to BERT, we investigate why scaling down is hard, and which modifications actually improve performance in this scenario. We provide evidence that even in this constrained setting, performance closely follows scaling laws observed in large-compute settings. Through the lens of scaling laws, we ...

The MiniPile Challenge for Data-Efficient Language Models

bookmark · Added on July 29, 2024 · 18 min read

The MiniPile Challenge introduces a new dataset for pre-training language models, containing 1 million documents filtered for quality. It aims to reduce the need for large computational resources while still achieving competitive performance on language tasks. The research shows that models pre-trained on MiniPile perform only slightly worse than those trained on much larger datasets.

Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget

bookmark · Added on July 29, 2024 · 1 min read

The authors present a method for training large text-to-image diffusion models on a very low budget. They use a technique called deferred masking to minimize performance loss while reducing computational costs. Their approach achieves high-quality results at a fraction of the cost compared to existing models, demonstrating the potential for democratizing AI training.

1024cores

bookmark · Added on July 29, 2024 · 1 min read

Dmitry Vyukov shares information on synchronization algorithms, multicore design patterns, and high-performance computing on his website, 1024cores.net. He focuses on shared-memory systems and does not cover topics like clusters or GPUs. New content is added regularly, and readers can subscribe for updates.

Implementing interactive languages

bookmark · Added on July 28, 2024 · 8 min read

Implementing an interactive language requires considering both compile-time and run-time performance. Traditional switch-based bytecode interpreters are easy to implement but have lower run-time performance compared to optimizing compilers. A sweet spot in performance can be found by aiming for combined compile-time and run-time performance within a certain range. Various options for implementing fast interpreters, existing compilers like LLVM and Cranelift, custom compilers, and using WebAssembly as a backend are discussed. The idea of having two backends for a language to support quick startup and aggressive optimization is also explored. There are still many unknowns and further research is needed to determine the feasibility and performance of different approaches.

Pointers Are Complicated, or: What's in a Byte?

bookmark · Added on July 28, 2024 · 14 min read

The document explains the complexities of pointers in low-level programming languages like C++ and Rust, debunking the misconception that pointers are simple integers. It delves into examples showing how assumptions about pointers can lead to undefined behavior and how pointer arithmetic can be tricky. The text proposes a model where a pointer is a pair of an allocation ID and an offset, rather than just an integer. Additionally, it discusses the challenges of representing bytes in memory, especially when dealing with uninitialized memory and the need for a more nuanced byte representation to ensure program correctness.

Three Architectures for a Responsive IDE

bookmark · Added on July 28, 2024 · 10 min read

The text discusses three architectures for a responsive IDE: indexing on a per-file basis, using a FQN index for completion, and a query-based compiler approach. Each approach has its own challenges and benefits, such as handling macro expansions and managing dependencies efficiently to ensure fast performance.

How a Zig IDE Could Work Feb 10, 2023

bookmark · Added on July 28, 2024 · 9 min read

The author discusses how to build an Integrated Development Environment (IDE) for the Zig programming language, which has unique features like a simple syntax but also complex compile-time evaluation. The IDE needs to handle incomplete code and provide immediate feedback while managing rapid code changes. The post explores various strategies for efficiently processing code, such as using abstract interpretation and optimizing compilation to focus only on necessary parts of the codebase.

Properly Testing Concurrent Data Structures Jul 5, 2024

bookmark · Added on July 28, 2024 · 23 min read

The article discusses how to effectively test concurrent data structures by using managed threads that can be paused and resumed. It explains the importance of controlling thread execution to avoid issues like race conditions while executing random operations. The author emphasizes the need for proper synchronization mechanisms to ensure that only one thread is active at a time during tests.

Parse, don’t validate

bookmark · Added on July 28, 2024 · 20 min read

The text discusses the importance of parsing over validating in Haskell to prevent errors and enhance code reliability by using strong argument types. Parsing upfront helps maintain consistency and avoids potential issues with partial input processing, demonstrating the benefits of type-driven design in Haskell programming. The text also touches on the subjective nature of programming languages, highlighting differing perceptions of Haskell and the challenges faced by learners in navigating diverse opinions.

Too Fast, Too Megamorphic: what influences method call performance in Java?

bookmark · Added on July 28, 2024 · 11 min read

The performance of method calls in Java can be improved through techniques like inlining and using inline caches. Monomorphic calls, where only one method can be invoked, are the fastest, while bimorphic and megamorphic calls are slower due to increased lookup costs. The study highlights that simply adding the "final" keyword or overriding methods does not significantly enhance performance.

The Black Magic of (Java) Method Dispatch

bookmark · Added on July 28, 2024 · 1 min read

The content shows code execution percentages for different operations within a program. It includes instructions for handling different coders, with comparisons and jumps based on coder values. The code includes sections like the main entry point, epilogue, handling other coders, and specific coder cases like Coder1 and Coder2.

Why null sucks, even if it's checked

bookmark · Added on July 28, 2024 · 14 min read

The article discusses the problems with using null in programming languages like Kotlin and C#, highlighting that null can lead to confusion and errors. It argues that null is not an extensible solution for representing absence of value and suggests using sum types or optional types instead. The author believes that languages should focus on improving optional types rather than trying to make null safer.

Unnamed Document

bookmark · Added on July 28, 2024 · 1 min read

Resources for Building Programming Languages

bookmark · Added on July 28, 2024 · 4 min read

The article shares resources for learning how to create programming languages, focusing on Rust and C. It highlights the book "Crafting Interpreters," which provides practical insights into building interpreters using different programming approaches. The author also discusses their personal experience building a language and the tools they've found helpful, like LLVM and Cranelift.

Little 'Big Ideas' in Programming Language Design

bookmark · Added on July 28, 2024 · 9 min read

Colin Davis discusses "little big ideas" in programming language design, focusing on the balance between innovative features and conventional choices. He highlights Mojo and Go as examples, noting how Mojo combines modern improvements with familiar concepts, while Go prioritizes simplicity and a strong ecosystem. Davis suggests that small design decisions, like memory management and parameter passing, can greatly enhance a language's usability and performance.

Computer Networking: A Top-Down Approach

bookmark · Added on July 27, 2024 · 2 min read

Jim Kurose and Keith Ross are prominent computer science professors with extensive experience in networking and related fields. They have received multiple awards for their teaching and research, and both have held leadership roles in academic and professional organizations. Their work focuses on topics like network protocols, security, and multimedia communication.

Using Uninitialized Memory for Fun and Profit Posted on Friday, March 14, 2008.

bookmark · Added on July 27, 2024 · 5 min read

A clever trick involves using uninitialized memory to improve performance in certain programming situations by representing sparse sets efficiently with two arrays that point at each other. This technique allows for fast constant-time operations for adding, checking, and clearing elements in the set, making it a valuable tool for optimizing algorithms and data structures. The sparse set representation is especially useful for scenarios where speed is critical, such as in compiler optimizations and graph traversal algorithms.

Zip Files All The Way Down

bookmark · Added on July 27, 2024 · 12 min read

The text discusses creating self-reproducing programs and files like zip files that can decompress to themselves. It explores using Lempel-Ziv compression for self-reproduction and the challenges of translating these concepts into real opcode encodings like DEFLATE used in gzip and zip files. The ultimate goal is to create a zip file that contains a larger copy of itself recursively, creating a chain of expanding zip files.

UTF-8: Bits, Bytes, and Benefits Posted on Friday, March 5, 2010.

bookmark · Added on July 27, 2024 · 4 min read

UTF-8 is a straightforward way to encode Unicode code points into a byte stream, and understanding its inner workings is key to leveraging its benefits. Key properties of UTF-8 include preserving ASCII files, ensuring ASCII bytes are represented as themselves, and requiring code points to be encoded using the shortest possible sequence. The encoding is self-synchronizing, facilitating substring searches and making it compatible with most programs that handle 8-bit files safely. While some tools may need modification to handle UTF-8, it is increasingly becoming the standard encoding due to its practical advantages and simple design.

Minimal Boolean Formulas

bookmark · Added on July 27, 2024 · 25 min read

The post discusses how to compute the minimum number of AND and OR operators needed for Boolean functions with five variables. It describes the author's program that efficiently calculates this minimum for various functions while also improving algorithms for speed. The findings contribute to understanding the complexity of Boolean functions and their representations.

Hacking the OS X Kernel for Fun and Profiles Posted on Tuesday, August 13, 2013.

bookmark · Added on July 27, 2024 · 9 min read

The article discusses a bug in the OS X kernel related to how profiling signals are delivered in multithreaded processes. It explains that the kernel incorrectly sends the SIGPROF signal to the entire process instead of the specific running thread. The author outlines a fix involving a small edit to the kernel code to ensure the signal is sent to the correct thread.

How To Build a User-Level CPU Profiler Posted on Thursday, August 8, 2013.

bookmark · Added on July 27, 2024 · 9 min read

The text discusses how the pprof tool simplifies CPU profiling for C++ and Go programs by utilizing hardware timers and the operating system. Profiling information is gathered through hardware interrupts, providing insights into a program's performance and resource usage. By moving profiling logic to user-level timers, programs can customize and enhance profiling capabilities without kernel changes.

An Encoded Tree Traversal

bookmark · Added on July 27, 2024 · 5 min read

The text discusses different ways to traverse binary trees and how these methods can be generalized to k-ary trees. It highlights a new ordering for traversing k-ary trees that results in a regular numbering pattern, which is not present in the traditional methods. The author seeks references or examples of this k-ary-coded traversal order, which he has not yet found.

Our Software Dependency Problem

bookmark · Added on July 27, 2024 · 26 min read

The text discusses the risks and benefits of using software dependencies in programming. It emphasizes the importance of understanding, managing, and monitoring dependencies to prevent potential issues like bugs and security vulnerabilities. The article highlights the need for developers to establish best practices for effectively utilizing dependencies in their projects.

The Magic of Sampling, and its Limitations Posted on Saturday, February 4, 2023.

bookmark · Added on July 27, 2024 · 9 min read

Sampling can help estimate the percentage of items with a specific trait accurately. The number of samples taken greatly affects the accuracy of the estimate. To get precise estimates, all items must have an equal chance of being selected during sampling.

Running the “Reflections on Trusting Trust” Compiler Posted on Wednesday, October 25, 2023.

bookmark · Added on July 27, 2024 · 24 min read

The text discusses how to modify a C compiler to insert a backdoor into a program without leaving traces in the source code. It explains that the backdoor can be detected because the compiler's size increases each time it compiles itself. Finally, it highlights the importance of using trusted compilers to prevent hidden backdoors in modern software development.

Improving the Font Pipeline

bookmark · Added on July 26, 2024 · 22 min read

To improve the font pipeline, consider how to efficiently choose and render glyphs for different languages, including handling ligatures and memory constraints. You may need to create texture atlases for various glyphs while ensuring new translations are incorporated. Finally, optimize rendering to avoid blurriness and ensure smooth performance across different character sets.

Easy Scalable Text Rendering on the GPU

bookmark · Added on July 26, 2024 · 6 min read

This text explains a fast and memory-efficient technique for rendering text on the GPU without using traditional methods like signed distance fields. It uses triangles to fill in pixels inside the glyph and supports subpixel anti-aliasing for crisp text on LCD screens. The technique is resolution-independent, simple to implement, and can be extended to enhance rendering quality.

Adventures in Text Rendering: Kerning and Glyph Atlases

bookmark · Added on July 26, 2024 · 18 min read

Text rendering involves converting vector glyphs to bitmaps, positioning them on screen, and optimizing performance by using glyph atlases. Glyph atlases store rasterized glyphs efficiently, allowing for sub-pixel alignment and improved rendering quality. This approach balances performance and quality in text rendering for different types of fonts.

Exploring the Power of Negative Space Programming

bookmark · Added on July 25, 2024 · 5 min read

Negative space programming helps improve code by defining what it should not do, making it more robust and clear. By using constraints and assertions, developers can catch errors early and enhance security. This approach also promotes simplicity, making the code easier to maintain and understand.

CompilerTalkFinal

bookmark · Added on July 25, 2024 · 19 min read

The content discusses various compilers and their features, including Clang, GCC, V8, CakeML, Chez Scheme, and more. It also touches on the history of interpreters and compilers, with examples like ENIAC and the first compiler developed by Grace Hopper. Different approaches to compilation and interpretation are highlighted, showcasing the evolution of compiler technology.

Graydon Hoare: 21 compilers and 3 orders of magnitude in 60 minutes

bookmark · Added on July 25, 2024 · 1 min read

Graydon Hoare's talk explains different approaches to building compilers, from traditional giants to more efficient variants. He highlights the importance of using compiler-friendly languages and theory-driven meta-languages. The presentation covers key concepts like sophisticated partial evaluation and implementing compilers directly by hand.

p75-hoare

bookmark · Added on July 25, 2024 · 41 min read

The author recounts experiences in designing a computer programming language and issues a warning about language complexity. Despite challenges, a subset of the language was successfully implemented. The author emphasizes the importance of simplicity and reliability in programming languages for critical applications.

Updating the Go Memory Model

bookmark · Added on July 23, 2024 · 20 min read

The Go memory model needs updates to clarify how synchronization works and to endorse race detectors for safer concurrency. It suggests adding typed atomic operations and possibly unsynchronized atomics to improve program correctness and performance. The goal is to ensure that Go programs behave consistently and avoid data races, making them easier to debug.

Programming Language Memory Models (Memory Models, Part 2) Posted on Tuesday, July 6, 2021. PDF

bookmark · Added on July 23, 2024 · 45 min read

Modern programming languages use atomic variables and operations to help synchronize threads and prevent data races. This ensures that programs run correctly by allowing proper communication between threads without inconsistent memory access. All major languages, like C++, Java, and Rust, support sequentially consistent atomics to simplify the development of multithreaded programs.

Hardware Memory Models (Memory Models, Part 1) Posted on Tuesday, June 29, 2021. PDF

bookmark · Added on July 23, 2024 · 27 min read

This text discusses hardware memory models, focusing on how different processors handle memory operations and maintain order. It explains the concept of sequential consistency, where operations are executed in a predictable order, and contrasts it with more relaxed models like those used in ARM and POWER architectures. The author highlights the importance of synchronization to avoid data races in concurrent programming.

Baby Steps to a C Compiler

bookmark · Added on July 23, 2024 · 3 min read

Writing a simple compiler can help you understand how computers work. Start with a minimal project that compiles a small subset of a language, and then gradually add more features. This approach makes learning about compilers and programming enjoyable and rewarding.

Kernel Programming Guide

bookmark · Added on July 23, 2024 · 1 min read

Essential information for programming in the OS X kernel. Includes a high-level overview.

Tiny Tapeout

bookmark · Added on July 23, 2024 · 1 min read

Tiny Tapeout is a project that helps people easily and affordably create their own chip designs. It offers resources for beginners and advanced users, along with a special price for submissions. Join the community to learn and share your designs before the deadline on September 6th.

Why Pascal is Not My Favorite Programming Language

bookmark · Added on July 23, 2024 · 31 min read

Pascal is not recommended for serious programming due to limitations in its standard form. The language's strict typing and lack of features like separate compilation make it challenging for complex projects. Pascal is better suited for educational purposes rather than practical programming tasks.

What Color is Your Function?

bookmark · Added on July 23, 2024 · 16 min read

Functions in a programming language can be either red or blue, affecting how they are called and used. Red functions are asynchronous and typically more complex to work with than blue functions. The choice between red and blue functions can impact code organization and maintainability.

What is an Invariant? Oct 6, 2023

bookmark · Added on July 22, 2024 · 9 min read

Invariants are properties that hold true during the evolution of a system, helping to ensure correct behavior in programming. They can simplify reasoning about code, whether it’s for small algorithms or larger systems. By clearly defining invariants, programmers can create robust code and manage complex systems effectively.

Chess-GPT's Internal World Model

bookmark · Added on July 22, 2024 · 17 min read

The blog post discusses how a GPT model trained on chess games learns to predict moves and track the board state without being explicitly given the rules. It successfully classified chess pieces with high accuracy and estimated player skill levels based on game moves. The findings suggest that models trained on strategic games can effectively learn complex tasks through pattern recognition.

Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models

bookmark · Added on July 22, 2024 · 1 min read

Researchers trained a chess-playing language model to understand the game without prior knowledge, focusing on how it represents the board state. They found that the model not only learned the board's layout but also estimated player skill, which helped it predict the next move better. By incorporating a player skill vector, the model's win rate improved significantly.

Manipulating Chess-GPT's World Model

bookmark · Added on July 22, 2024 · 15 min read

The author explores how Chess-GPT, a language model for chess, can improve its performance by manipulating its internal understanding of player skill and board state. By using linear probes and skill interventions, the model's chess-playing ability was significantly enhanced, especially in games with random initializations. The findings suggest that Chess-GPT learns a deeper understanding of chess rather than just memorizing patterns.

Crafting an Interpreter in Zig - part 1

bookmark · Added on July 22, 2024 · 3 min read

The author is learning Zig by implementing an interpreter for the Lox programming language, inspired by the book "Crafting Interpreters." They are documenting their journey, focusing on interesting aspects of Zig and how it differs from C. So far, they have enjoyed the process, particularly the simplicity and power of Zig's generic programming.

Teach Yourself Programming in Ten Years

bookmark · Added on July 21, 2024 · 13 min read

The text discusses the misconception of quickly learning programming in a short time, emphasizing that true expertise takes about ten years of dedicated practice and learning. It highlights the importance of hands-on experience, interacting with other programmers, and working on various projects to become a skilled programmer. The text emphasizes that mastering programming requires practical application, learning from others, and continuous practice over time.

What Every Computer Scientist Should Know About Floating-Point Arithmetic

bookmark · Added on July 21, 2024 · 2h 45m read

The text discusses the challenges and considerations of floating-point arithmetic in computer science. It emphasizes the importance of rounding in floating-point calculations and the implications of different precision levels. Additionally, it highlights the need for careful implementation to ensure correctness and accuracy in programs that rely on floating-point arithmetic.

The Development of the C Language*

bookmark · Added on July 20, 2024 · 43 min read

The paper discusses the development and influences of the C programming language, highlighting its creation at Bell Labs and transition from the B language. C's simplicity, efficiency, and widespread adoption across various platforms and architectures are emphasized, showcasing its enduring stability and usefulness in software development. Despite its quirks and historical origin, C has proven to be a powerful and versatile language for programmers worldwide.

Class Warfare

bookmark · Added on July 20, 2024 · 5 min read

The text discusses a woman's conversation about company politics and self-interest, highlighting a zero-sum mentality within organizations. It emphasizes the need to shift away from this mindset and focus on creating value instead. The author suggests that combating this mentality starts with internal change and encourages individuals to reject zero-sum thinking for long-term benefit.

Ownership

bookmark · Added on July 20, 2024 · 1 min read

A Note About Zig Books for the Zig Community

bookmark · Added on July 18, 2024 · 8 min read

The author discusses the idea of writing a Zig book and shares personal plans for self-publishing their own book. They weigh the pros and cons of working with a publisher versus self-publishing, emphasizing the importance of considering creative freedom and revenue sharing. The author encourages those interested in writing a Zig book to carefully evaluate their options, noting that the Zig community values learning materials and support.

Your Starting Point!

bookmark · Added on July 17, 2024 · 25 min read

The text discusses the concepts of three-dimensional objects and how they are represented in two dimensions for computer graphics. It explains the process of projecting 3D points onto a canvas to create images. The importance of geometry and mathematics in computer graphics, particularly in defining objects and creating images, is emphasized.

Zig Interfaces for the Uninitiated, an update

bookmark · Added on July 17, 2024 · 6 min read

The post discusses a new idiom for runtime polymorphism in Zig, focusing on using fat pointers instead of @fieldParentPtr. It provides a step-by-step guide on creating a formal Iterator interface and implementing it with an example range iterator. The drawbacks of this pattern include potential performance issues and the requirement for the original implementor to remain alive for the interface to function correctly.

Zig Interfaces for the Uninitiated

bookmark · Added on July 17, 2024 · 8 min read

The text discusses how to create and implement generic iterators in Zig using interfaces like `Iterator` and `Range`. It demonstrates how to use these iterators to iterate over ranges of values and provides examples of ascending, descending, and skipping ranges. Additionally, it introduces a function `fold` to apply a function to successive elements in an iterator, showcasing Zig's runtime polymorphism for data structures.

Exploring Compile-Time Interfaces in Zig

bookmark · Added on July 17, 2024 · 7 min read

Zig is a programming language with active community support and a focus on efficient, reusable software development. Interfaces in Zig define a blueprint for classes to implement specific methods, promoting code abstraction and flexibility. Compile-time interfaces in Zig optimize code structure by resolving methods during compilation for efficient program execution.

Aro - a C compiler

bookmark · Added on July 17, 2024 · 1 min read

Aro is a C compiler created as an alternative to Zig's compiler. It includes the aro module for the compiler and a language-agnostic aro_backend module for translating code into machine code. Aro uses self-hosted backends from the Zig compiler for optimization.

Database Systems

bookmark · Added on July 17, 2024 · 1 min read

This course at CMU covers database management systems, including data models, query languages, storage architectures, and more. It uses case studies to show real-world applications and is suitable for students with basic systems programming skills. The course also thanks companies for their support in equipment donations and course development.

Discovering and exploring mmap using Go

bookmark · Added on July 17, 2024 · 7 min read

Memory-mapped files allow programs to access disk data larger than available memory. By using mmap in Go, you can map a file directly into memory for easier manipulation. Virtual memory techniques, like mmap, can help solve memory limitations in handling large files efficiently.

But how, exactly, databases use mmap?

bookmark · Added on July 17, 2024 · 10 min read

Databases use memory-mapped files like mmap to handle data on disk larger than available memory. Examples include SQLite, LevelDB, Lucene, LMDB, and MongoDB. By understanding how mmap is used, we can grasp how databases efficiently read and write data from disk.

reHow memory mapped files, filesystems and cloud storage works

bookmark · Added on July 17, 2024 · 2 min read

Kelly discusses the challenges of memory-mapped files and cloud storage in response to a comment about space reservation in Voron. Cloud providers may allocate more space than needed, leading to unexpected charges and unreliable data handling. Testing reveals issues with sparse files and memory mapping in cloud scenarios, highlighting the importance of understanding storage limitations.

Implementing a file pager in Zig

bookmark · Added on July 17, 2024 · 5 min read

Implementing a file pager in Zig involves delaying disk writes until a threshold is reached. Two eviction strategies include least recently used and least frequently used models. Prioritizing pages based on usage can help optimize performance.

Criticizing Hare language approach for generic data structures

bookmark · Added on July 17, 2024 · 5 min read

The blog criticizes the Hare language approach for not providing generic data structures like hash maps in its standard library. It highlights the complexity and importance of hash tables in various programming languages and emphasizes the need for efficient data structures in modern programming ecosystems. The author disagrees with Hare's approach and stresses the significance of hash tables in software development.

spikedoanz/from-bits-to-intelligence: machine learninig stack in under 100,000 lines of code

bookmark · Added on July 16, 2024 · 2 min read

The text discusses building a machine learning stack in under 100,000 lines of code with hardware, software, tensors, and machine learning components. It outlines the required components like a CPU, GPU, storage, C compiler, Python runtime, operating system, and more. The goal is to simplify the machine learning stack while providing detailed steps for implementation in different programming languages.

One year of C

bookmark · Added on July 16, 2024 · 8 min read

The author reflects on their year of writing C code, finding it enjoyable and productive. They emphasize the importance of choosing the right language for each problem and share insights on the benefits of using C over C++ in certain scenarios. Additionally, they discuss the advantages of C99 improvements and the simplified nature of writing C code compared to C++.

Heap Memory and Allocators

bookmark · Added on July 15, 2024 · 22 min read

The text discusses different types of memory allocators in Zig programming language. It explains how memory allocation and deallocation work using alloc and free functions. Various allocator types like GeneralPurposeAllocator and FixedBufferAllocator are highlighted for managing memory efficiently.

Pointers

bookmark · Added on July 15, 2024 · 16 min read

Emulator 101

bookmark · Added on July 15, 2024 · 1 min read

A detailed, step by step guide to writing an emulator

Data Compression Explained

bookmark · Added on July 15, 2024 · 4h 4m read

Data compression involves modeling and coding to reduce the size of data files. Modern compressors typically use arithmetic coding for efficient compression. Algorithms like Huffman coding and run-length encoding are commonly used to achieve better compression results.

Twitter's Recommendation Algorithm

bookmark · Added on July 14, 2024 · 7 min read

Twitter uses a recommendation algorithm to select the top tweets for users' timelines. The algorithm is based on core models and features that extract information from tweet, user, and engagement data. The recommendation pipeline consists of three main stages: candidate sourcing, ranking, and applying heuristics and filters. Twitter uses both in-network and out-of-network sources to find relevant tweets, and employs embedding spaces to determine content similarity. The final step involves blending tweets with other non-tweet content before sending them to users' devices. The goal of Twitter's open source endeavor is to provide transparency to users about how the recommendation system works.

Programming languages resources

bookmark · Added on July 13, 2024 · 3 min read

This page is a collection of the author's favorite resources for people getting started writing programming languages. The resources cover various aspects such as compilers, runtimes, runtime optimization, pointer tagging, JIT compilers, assembler libraries, and interesting tools. The author also mentions topics they want to write about in the future and papers they want to read. The page is meant to be a helpful reference for those interested in programming language implementation.

3D Math Primer for Graphics and Game Development

bookmark · Added on July 12, 2024 · 1 min read

The book "3D Math Primer for Graphics and Game Development" is available to read for free on the gamemath.com website. It includes information about GDC talks, FAQs, and resources for the first edition of the book. The first edition, published in 2002, is described as high tech, but the author recommends reading the second edition instead, which is also available for free.

Welcome to OpenGL

bookmark · Added on July 12, 2024 · 3 min read

This text is about learning modern OpenGL through an online book that covers basic, intermediate, and advanced knowledge with clear examples and practical concepts. The content is freely available online and in print, with the aim of providing a complete and easy-to-understand platform for graphics programming enthusiasts. Readers will learn core graphics aspects, useful techniques, and even create a small game based on the obtained OpenGL knowledge.

WebGPU Fundamentals

bookmark · Added on July 12, 2024 · 1 min read

The text provides a collection of articles to help beginners learn the basics of WebGPU, covering topics like fundamentals, 3D math, lighting techniques, and compute shaders. It also includes information on optional features, data memory layout, transparency, performance, and resources for further learning. Readers can explore various aspects of WebGPU, including how it works, 2D and 3D techniques, and essential concepts like uniforms, textures, and storage buffers.

An opinionated beginner’s guide to Haskell in mid-2019

bookmark · Added on July 12, 2024 · 17 min read

This guide is for beginners in Haskell or those transitioning from similar languages, offering advice on learning resources and tools. It emphasizes the importance of writing Haskell code, getting help online, choosing popular platforms, and sticking to the default Prelude. The guide also touches on application architecture, using records, debugging techniques, and the experimental nature of Haskell as both a research and industrial language.

Are tagged unions overrated?

bookmark · Added on July 12, 2024 · 11 min read

The author discusses the limitations of tagged unions and pattern matching in language development, suggesting that they are overrated for implementing language ASTs and IRs. Despite the benefits of tagged unions, the complexity they add may not always justify their use, especially in cases where simpler alternatives like class hierarchies can offer similar functionality. The post also highlights the potential for enhancing pattern-matching capabilities in mainstream languages to improve code readability and maintainability.

C++ Core Guidelines

bookmark · Added on July 12, 2024 · 9h 40m read

These guidelines aim to simplify and improve the safety of C++ code by recommending specific extensions and best practices. They focus on static type safety, resource management, and reducing the likelihood of errors or accidents. By following these guidelines, programmers can write more correct, safer code without sacrificing performance.

What every systems programmer should know about concurrency

bookmark · Added on July 11, 2024 · 32 min read

The document delves into the complexities of concurrency for systems programmers, explaining the challenges of running multithreaded programs where code is optimized and executed in unexpected sequences. It covers fundamental concepts like atomicity, enforcing order in multithreaded programs, and memory orderings. The text emphasizes the importance of understanding how hardware, compilers, programming languages, and applications interact to create a sense of order in multithreaded programs. Key topics include atomic operations, read-modify-write operations, compare-and-swap mechanisms, and memory barriers in weakly-ordered hardware architectures.

compiler_construction

bookmark · Added on July 11, 2024 · 47 min read

Building a compiler can be straightforward by breaking the development into small steps and using Scheme as the implementation language. The tutorial focuses on translating a subset of Scheme to assembly code, with a step-by-step approach to achieve a fully working compiler. Testing and refining the compiler incrementally leads to a powerful tool capable of compiling an interactive evaluator.

How do we tell truths that might hurt?

bookmark · Added on July 11, 2024 · 3 min read

The document discusses the challenges of telling unpleasant truths and the conflict that arises when sharing these truths in the field of Computing Science. The author argues that remaining silent about these truths compromises the intellectual integrity of the field. The document also lists a number of truths related to programming languages and the use of language in computing systems. The author questions whether the field should continue to ignore these truths and urges for a change in attitude.

The next fifty years

bookmark · Added on July 11, 2024 · 10 min read

The text discusses the future of computing science over the next fifty years, emphasizing the importance of simplicity and elegance in design to prevent complexity. It highlights the close connection between program design and proof design, suggesting that advancements in program design can impact general mathematics. The author encourages embracing the opportunity to simplify processes and design systems that rely on formal mathematics.

Recommender Systems: A Primer

bookmark · Added on July 10, 2024 · 2h 28m read

Personalized recommendations have become a common feature of modern online services, including most major e-commerce sites, media platforms and social networks. Today, due to their high practical relevance, research in the area of recommender systems is flourishing more than ever. However, with the new application scenarios of recommender systems that we observe today, constantly new challenges arise as well, both in terms of algorithmic requirements and with respect to the evaluation of such systems. In this paper, we first provide an overview of the traditional formulation of the recommendation problem. We then review the classical algorithmic paradigms for item retrieval and ranking and elaborate how such systems can be evaluated. Afterwards, we discuss a number of recent developments in recommender systems research, including research on session-based recommendation, biases in recommender systems, and questions regarding the impact and value of recommender systems in practice.

http client in the standard library · Issue #2007 · ziglang/zig

bookmark · Added on July 10, 2024 · 9 min read

The issue #2007 discusses the implementation of an HTTP client in Zig's standard library. Contributors debate the necessity and scope of including an HTTP client, considering factors like complexity and resource allocation. Ultimately, the HTTP client implementation was completed and closed as part of milestone 0.12.0.

Introduction to Compilers and Language Design

bookmark · Added on July 10, 2024 · 1 min read

A compiler translates high-level code to lower-level code, and building one is a common project in computer science education. This book provides a beginner-friendly guide to building a compiler for a C-like language, suitable for undergraduates with programming experience. The author offers free online access to the textbook and related code resources, with options to purchase a physical copy.

Bare Metal Zig

bookmark · Added on July 10, 2024 · 10 min read

The text discusses compiling a freestanding Zig binary to run on "bare metal" without relying on an operating system. It shows how to create a simple freestanding binary, make it multiboot compliant, and add custom console functionality for output. The process involves targeting specific architectures, handling linker warnings, and ultimately creating a bootable "kernel" to run on virtual machines like QEMU.

Comparing SIMD on x86-64 and arm64

bookmark · Added on July 10, 2024 · 1h 9m read

The text compares SIMD implementations using SSE on x86-64 and Neon on arm64 processors, including emulating SSE on arm64 with Neon. It explores vectorized code performance using intrinsics, auto-vectorization, and ISPC, highlighting the efficiency of SSE and Neon implementations. The study shows how optimizing for SIMD instructions significantly boosts performance over scalar implementations in ray-box intersection tests.

Compiler Optimizations Are Hard Because They Forget

bookmark · Added on July 10, 2024 · 11 min read

Compiler optimizations involve breaking down complex changes into smaller, more manageable steps to improve code efficiency. However, as more optimizations are added, the potential for errors and missed opportunities increases, making it challenging to maintain optimal performance. Compilers struggle with balancing aggressive optimizations while preserving correct program behavior, highlighting the complexity and difficulties inherent in optimizing compilers.

C Isn't A Programming Language Anymore

bookmark · Added on July 10, 2024 · 19 min read

C is no longer just a programming language but a vital protocol for all languages. Parsing C headers is a complex task best left to C compilers. Maintaining ABI compatibility in C can be challenging and may require versioning schemes.

Writing a C Compiler, Part 1

bookmark · Added on July 9, 2024 · 18 min read

This text is about creating a C compiler in multiple stages, starting with lexing, parsing, and code generation. The process involves breaking down the source code, building an abstract syntax tree, and generating x86 assembly code. The compiler will handle simple programs with a single main function and a return statement.

GitHub - DoctorWkt/acwj: A Compiler Writing Journey

bookmark · Added on July 9, 2024 · 2 min read

This GitHub repository documents the author's journey to create a self-compiling compiler for a subset of the C language. The author shares steps taken and explanations to help others follow along practically. The author credits Nils M Holm's SubC compiler for inspiration and differentiates their code with separate licensing.

A new JIT engine for PHP-8.4/9

bookmark · Added on July 9, 2024 · 1 min read

A new JIT engine for PHP is being developed, improving performance and simplifying development. The engine will be included in the next major PHP version, potentially PHP 9.0. The new JIT engine generates a single Intermediate Representation (IR), eliminating the need to support assembler code for different CPUs.

Unknown

bookmark · Added on July 9, 2024 · 1h 30m read

Hardware prefetching in multicore processors can be too aggressive, wasting resources and impacting performance for co-running threads. Combining hardware and software prefetching can optimize performance by efficiently handling irregular memory accesses. A method described in Paper II offers a low-overhead framework for accurate software prefetching in applications with irregular access patterns.

Introduction 2016 NUMA Deep Dive Series

bookmark · Added on July 9, 2024 · 3 min read

The 2016 NUMA Deep Dive Series by staroceans.org explores various aspects of computer architecture, focusing on NUMA systems and their optimization for performance. The series covers topics such as system architecture, cache coherency, memory optimization, and VMkernel constructs to help readers understand and improve their host design and management. The series aims to provide valuable insights for configuring and deploying dual socket systems using Intel Xeon processors, with a focus on enhancing overall platform performance.

von Neumann architecture - Wikipedia

bookmark · Added on July 9, 2024 · 20 min read

The von Neumann architecture is a computer design with a processing unit, control unit, memory, and input/output mechanisms. It allows for instructions and data operations to be stored in memory, advancing computer technology from fixed-function machines like the ENIAC. This architecture was influenced by the work of Alan Turing and John von Neumann and has been widely used in the development of modern computers.

Compiling tree transforms to operate on packed representations

bookmark · Added on July 8, 2024 · 1 min read

The article explains how tree traversals in programming can be optimized by compiling them to work on serialized tree structures without using pointers. This approach can make programs run significantly faster on current x86 architectures. The authors developed a prototype compiler for a functional language that generates efficient code for traversing trees using packed data representations.

Pipelines Support Vectorized, Point-Free, and Imperative Style

bookmark · Added on July 8, 2024 · 4 min read

The text discusses how pipelines in the shell language support vectorized operations on collections and point-free style, where no data is explicitly mentioned. It also demonstrates how imperative code can be incorporated within pipelines for tasks like generating HTML tables. The unique features of pipelines include their ability to handle vectorized code, point-free composition, and integration of imperative instructions.

Entering text in the terminal is complicated

bookmark · Added on July 8, 2024 · 9 min read

Entering text in the terminal can be challenging due to inconsistencies in how different programs handle text input. Some programs support basic features like arrow keys and history navigation, while others have custom input systems with advanced functionalities. Understanding the input mode of a program can help users navigate text editing more effectively in the terminal.

What happens when you start a process on Linux?

bookmark · Added on July 8, 2024 · 5 min read

The process of starting a new program on Linux involves using the fork and exec system calls. Fork creates a clone of the current process, while exec replaces that clone with the new program to be executed. The new process inherits most attributes from its parent, with memory being shared through copy-on-write to optimize performance.

Debug your programs like they're closed source!

bookmark · Added on July 8, 2024 · 7 min read

The author discusses debugging programs without looking at the source code by using system calls like open, execve, and write. System calls allow you to understand and monitor a program's behavior without needing access to its source code. By learning and utilizing system calls, you gain debugging superpowers that are platform-independent and useful for closed-source programs.

How I got better at debugging

bookmark · Added on July 8, 2024 · 4 min read

Julia Evans shares her journey of improving her debugging skills through logical thinking, confidence, expanding knowledge, communication, and using tools like strace and tcpdump. By being systematic, confident, knowledgeable, and open to collaboration, she transformed debugging from a challenging task to an exciting learning opportunity. Her story emphasizes the importance of persistence, curiosity, and practical problem-solving in mastering the art of debugging.

Media Page Under Construction

bookmark · Added on July 8, 2024 · 1 min read

Handmade Cities' media page is under construction, with some recordings missing. The videos from Handmade Boston 2023 have poor audio quality due to using a third-party A/V company. Freya's Masterclass footage was lost, and an abridged version will be shown at Dutch Game Day.

Infographics: Operation Costs in CPU Clock Cycles

bookmark · Added on July 8, 2024 · 23 min read

The text discusses the operation costs in CPU clock cycles for different types of operations, including simple operations, floating-point operations, and vector operations. It highlights that memory involvement can significantly impact operation costs, with some operations taking as little as 1 CPU cycle. Different CPU architectures and types of operations can result in varying costs, with some operations requiring specialized CPU support to work efficiently.

Handles are the better pointers

bookmark · Added on July 8, 2024 · 14 min read

The text discusses using 'index-handles' instead of raw or smart pointers for memory management in C and C++. It suggests centralizing memory management into systems, grouping items into arrays, and converting handles to pointers only when necessary. By following specific rules, such as not storing pointers and using handle-to-pointer conversion, memory safety and efficient memory usage can be maintained.

You're Not Sick of Programming

bookmark · Added on July 8, 2024 · 1 min read

Many people feel tired of programming and dream of quitting for a more fulfilling career, like farming or traveling. However, the real issue might be frustration with office politics, lack of product vision, and burnout rather than a true dislike of programming. Taking a break or addressing these underlying problems could help rediscover the creative potential of programming.

Zig Bare Metal Programming on STM32F103 — Booting up

bookmark · Added on July 8, 2024 · 15 min read

The text explains how to program the STM32F103 microcontroller using the Zig programming language. It covers topics such as memory layout, linker scripts, and compiling code for embedded systems. By following the provided instructions, readers can successfully compile and run their first embedded program on the microcontroller.

OWASP Top Ten

bookmark · Added on July 7, 2024 · 3 min read

The OWASP Top 10 is a guide for developers to understand critical security risks in web applications. Companies are encouraged to follow this document to improve the security of their web applications. The 2021 update includes new categories and ranking changes based on testing data and industry feedback.

Introduction

bookmark · Added on July 7, 2024 · 1 min read

The OWASP Cheat Sheet Series offers valuable security information on application security topics. Created by experts, these concise cheat sheets aim to provide easy-to-read security guidance. You can download the cheat sheets from this site and stay updated through the ATOM feed.

The Copenhagen Book

bookmark · Added on July 7, 2024 · 1 min read

The Copenhagen Book is a free and open-source guide for implementing auth in web applications. It is community-maintained and can be used alongside the OWASP Cheat Sheet Series. Suggestions or concerns can be addressed by opening a new issue.

Undefined Behavior deserves a better reputation

bookmark · Added on July 6, 2024 · 12 min read

Undefined Behavior is often viewed negatively, but it can be a valuable tool for language designers. It allows programmers to convey insights to the compiler for optimizations. Responsible use of Undefined Behavior can enhance language design and code performance.

KHM+15

bookmark · Added on July 6, 2024 · 52 min read

The text discusses a formal C memory model that supports integer-pointer casts, essential for low-level C programming. It proposes a quasi-concrete memory model that allows standard compiler optimizations while fully supporting integer-pointer casts. This model helps verify programs and optimizations that are challenging to validate with integer-pointer casts.

Learning LLVM (Part-1) - Writing a simple LLVM pass

bookmark · Added on July 5, 2024 · 2 min read

This text introduces learning about LLVM and writing LLVM passes, which are used for transforming or analyzing a program's intermediate representation. LLVM offers a versatile compiler infrastructure with modules like the frontend, middle-end, and backend for optimizing and generating machine-specific code. By understanding LLVM concepts and pass managers, developers can create efficient passes for tasks like performance optimization and code analysis.

Some Were Meant for C

bookmark · Added on July 5, 2024 · 1h 11m read

The document "Some Were Meant for C" explores the enduring significance of the C programming language, highlighting its dual role as both an application and systems programming language. It challenges common assumptions about C, emphasizing its unique communicative design that differs from managed languages. The document argues that C's explicit representations and memory access foster effective system-building and communication, making it a preferred choice for certain technical challenges. Additionally, it critiques the prevailing discourse that demonizes C, advocating for a nuanced understanding of its role in the programming landscape.

Xv6, a simple Unix-like teaching operating system

bookmark · Added on July 5, 2024 · 3 min read

Xv6 is a teaching operating system developed by MIT for their operating systems course. It is based on Unix V6, written in ANSI C, and runs on Intel x86 machines. The xv6 source code is available on GitHub and is used in lectures to teach operating system concepts.

C Is Not a Low-level Language

bookmark · Added on July 5, 2024 · 17 min read

C is often considered a low-level language, but this article argues that it is not. The author explains that vulnerabilities like Spectre and Meltdown occurred because processor architects were trying to build fast processors that exposed the same abstract machine as a PDP-11, which C programmers believe is close to the underlying hardware. However, the reality is that C code runs on a complex compiler that performs intricate transformations to achieve the desired performance. The article also discusses how C's memory model and optimizations make it difficult to understand and can lead to undefined behavior. The author suggests that instead of trying to make C code fast, it may be time to explore programming models on processors designed for speed.

Should you learn C to "learn how the computer works"?

bookmark · Added on July 5, 2024 · 12 min read

The author discusses whether learning C is necessary to understand how computers work, ultimately concluding that C is not a direct representation of computer operations. Learning C can still be beneficial for understanding computing concepts and history, but it operates within a virtual machine and abstracts certain hardware details. By learning C, you can gain insight into the relationship between programming languages, hardware, and the historical development of computing.

A Guide to Undefined Behavior in C and C++, Part 1

bookmark · Added on July 5, 2024 · 17 min read

The text explains that undefined behavior in C and C++ can lead to unpredictable program outcomes. Compilers may optimize code by exploiting undefined behavior, potentially causing programs to misbehave. It is important for programmers to understand how undefined behavior can impact program execution.

Using neural nets to recognize handwritten digits

bookmark · Added on July 5, 2024 · 1h 11m read

Neural networks can recognize handwritten digits by learning from examples. Sigmoid neurons play a key role in helping neural networks learn. Gradient descent is a common method used for learning in neural networks.

When Network is Faster than Cache

bookmark · Added on July 5, 2024 · 12 min read

Firefox introduced a feature called RCWN to improve web performance by racing cached requests against the network. In some cases, the network can be faster than fetching data from the cache due to various factors like browser bugs and resource prioritization. Factors like device hardware and the total number of assets served from the cache impact cache retrieval performance significantly.

John Carmack on Functional Programming in C++

bookmark · Added on July 5, 2024 · 12 min read

Functional programming in C++ can help in writing better software by making code easier to reason about and eliminating thread race conditions. Pure functions, which only rely on input parameters and produce consistent outputs, offer benefits such as thread safety and easier testing. Refactoring towards purity can improve code quality, even if full purity is not achieved, by disentangling computation from the environment it operates in.

Zig-style generics are not well-suited for most languages

bookmark · Added on July 5, 2024 · 12 min read

Zig-style generics, like those in C++, may not work well for all languages due to limitations in compiler support and type inference. Armchair suggestions about adopting Zig-style generics in other languages may overlook these challenges. The flexibility and metaprogramming capabilities in Zig may not easily translate to other statically-typed languages.

WebGL2 vs WebGL1

bookmark · Added on July 4, 2024 · 19 min read

WebGL is a 3D API that works as a rasterization engine, requiring users to provide code for rendering points, lines, and triangles. Users must create vertex and fragment shaders to control how WebGL processes and displays graphics. The WebGL API simplifies rendering by executing user-created functions to draw basic shapes like triangles.

WebGL How It Works

bookmark · Added on July 4, 2024 · 10 min read

The text explains how WebGL processes vertices to create triangles and render them with pixels using shaders. Varyings are used to pass data from the vertex shader to the fragment shader for color interpolation. Buffers are essential for transferring vertex data to the GPU for rendering, and attribute locations are assigned to specify how to extract and use this data efficiently.

The_Night_Watch

bookmark · Added on July 4, 2024 · 13 min read

The text discusses the importance of systems programmers in dealing with complex technical challenges, emphasizing their unique skills in debugging and problem-solving. It contrasts the roles of systems programmers with other computer professionals like GUI designers and PHP developers, highlighting the critical nature of systems programming in challenging scenarios. The text humorously portrays the intense and sometimes absurd experiences of systems programmers, showcasing their indispensable role in addressing technical issues efficiently and effectively.

FreeType

bookmark · Added on July 4, 2024 · 1 min read

FreeType is a software library for rendering fonts, available for free. It is designed to be small, efficient, and capable of producing high-quality font images. Users can find installation instructions, documentation, and ways to communicate with the FreeType team on their website.

A Freestanding Rust Binary

bookmark · Added on July 3, 2024 · 18 min read

To create a freestanding Rust executable for operating system development, we need to disable linking to the standard library and define our own entry point function. By compiling for a bare metal target like thumbv7em-none-eabihf, we can avoid linker errors and run Rust code without an underlying operating system. Additional linker arguments are required for specific operating systems like Linux, Windows, and macOS to resolve linker errors and build the freestanding Rust binary successfully.

Manually linking Rust binaries to support out-of-tree LLVM passes

bookmark · Added on July 3, 2024 · 7 min read

LLVM is a compiler infrastructure used by frontends like rustc to generate machine code. To add custom LLVM passes to a Rust binary, extra flags can be used during compilation to produce LLVM-IR and then link the binary properly using LLVM tools. By understanding how Rust's static libraries work and leveraging cargo for dependency management, custom LLVM passes can be integrated into Rust binaries efficiently.

The Rust Reference

bookmark · Added on July 3, 2024 · 7 min read

The Rust compiler can generate different types of output artifacts, such as runnable executables, Rust libraries, dynamic libraries, and static system libraries. Dependencies between crates can be linked in various formats, such as rlib and dynamic library formats, following specific rules set by the compiler. Understanding how to specify output formats like --crate-type=bin or --crate-type=lib can help control the compilation process for Rust crates, while also considering options for linking C runtimes dynamically or statically based on target features.

Rust Compiler Development Guide

bookmark · Added on July 3, 2024 · 14 min read

The Rust compiler processes and transforms your code for compilation. It uses different stages like lexing, parsing, and abstract syntax tree lowering. The compiler aims for correctness, performance, and supporting incremental compilation.

How to speed up the Rust compiler one last time

bookmark · Added on July 3, 2024 · 9 min read

The author at Mozilla is concluding their work on speeding up the Rust compiler after several years of dedicated effort. They wrote multiple blog posts detailing their performance optimizations and shared valuable lessons learned from the process. The author expressed gratitude to those who supported their work and highlighted the importance of ongoing contributions to Rust's development.

How to speed up the Rust compiler in March 2024

bookmark · Added on July 3, 2024 · 4 min read

In March 2024, updates on the Rust compiler's performance highlighted several key improvements. Changes like using a single codegen unit, marking Debug::fmt methods with #[inline], introducing a cache, and upgrading LLVM versions led to notable reductions in wall-time, binary size, and hash table lookups. Additionally, the availability of the Cranelift codegen backend for x86-64/Linux and ARM/Linux offers an alternative for faster compile times. While the author didn't contribute to speed improvements this time, overall performance from August 2023 to March 2024 showed reductions in wall-time, peak memory usage, and binary size, indicating steady progress in enhancing the Rust compiler's efficiency.

Zig Bits 0x4: Building an HTTP client/server from scratch

bookmark · Added on July 3, 2024 · 19 min read

The text explains how to create an HTTP client and server from scratch using Zig >=0.11. For the client, you need to set up requests, headers, and wait for responses. The server part involves defining functions to handle requests and running the server to accept connections.

Do We Really Need A Link Step?

bookmark · Added on July 3, 2024 · 2 min read

The author questions the need for a link step in native-code compilation for faster performance. They propose a "zero-link" approach where compilers directly write object code into the final executable file. This method could improve efficiency by avoiding unnecessary object files and incorporating symbol resolution within the executable itself.

Death Note: L, Anonymity & Eluding Entropy

bookmark · Added on July 2, 2024 · 52 min read

The text discusses Light's mistakes in using the Death Note and how they led to his de-anonymization by L. Light's errors, such as revealing his precise killing methods and using confidential police information, significantly reduced his anonymity. The text also explores strategies Light could have employed to better protect his anonymity while using the Death Note.

jamiebuilds/the-super-tiny-compiler: :snowman: Possibly the smallest compiler ever

bookmark · Added on July 2, 2024 · 1 min read

The Super Tiny Compiler is a simplified example of a modern compiler using easy-to-read JavaScript. It helps you understand how compilers work from start to finish. Compilers play a big role in the tools we use daily.

5 Days to Virtualization: A Series on Hypervisor Development

bookmark · Added on July 2, 2024 · 5 min read

A series on hypervisor development for Intel processors with virtualization support will be published next week, covering topics like setting up a test environment, driver skeleton creation, and multi-processor initialization. The series aims to aid new readers in building, testing, and understanding type-2 hypervisor development using C programming language. Recommended reading and detailed explanations will be provided to enhance knowledge and understanding of virtualization concepts.

In-depth analysis on Valorant’s Guarded Regions

bookmark · Added on July 2, 2024 · 14 min read

The text discusses how Valorant's anti-cheat system, Vanguard, uses innovative techniques to protect against memory manipulation by whitelisting threads and creating shadow regions. These methods involve cloning and modifying the game's paging tables to allow access to hidden memory without affecting performance. By implementing these advanced security measures, Vanguard effectively prevents cheats from bypassing its guarded regions.

Exploit Development: No Code Execution? No Problem! Living The Age of VBS, HVCI, and Kernel CFG

bookmark · Added on July 2, 2024 · 1h 31m read

The text discusses various techniques used in exploit development, particularly focusing on targeting the Windows kernel. It mentions concepts like Hypervisor-Protected Code Integrity (HVCI) and how exploits can manipulate memory to execute attacker-controlled code in kernel mode. The text also delves into details like leaking kernel-mode memory, constructing ROP chains on the kernel-mode stack, and utilizing functions like NtQuerySystemInformation to escalate privileges and perform malicious actions in the system.

Reader

bookmark · Added on July 2, 2024 · 6 min read

The Reader API by jina.ai helps extract clean, LLM-friendly text from web content, ensuring high-quality input for AI systems like agents and RAG. It can also search the web for the latest information to keep LLMs up-to-date, improve factuality, and reduce misinformation. Additionally, Reader can read images on webpages and PDFs, providing alt text for images and lightning-fast PDF processing, all available for free with flexible rate limits.

CheerpX versus WebContainers

bookmark · Added on July 2, 2024 · 2 min read

CheerpX is a client-side virtualization technology for running x86 executables and operating systems in the browser without modifications or recompilation. It offers cost-effective, secure, and private execution of native code, making it suitable for various web-based applications. CheerpX stands out from other solutions by supporting any x86 executable and providing a robust two-tier emulator for efficient code execution.

Creating a Rootkit to Learn C

bookmark · Added on July 2, 2024 · 32 min read

The text demonstrates creating a userland rootkit in C to hide malicious activities like network connections and files. By hooking into system calls like access() and write(), the rootkit can manipulate userland programs and evade detection by tools like netstat. The rootkit uses shared library injections and hooks to intercept and manipulate system calls, showcasing the power of C for malicious activities.

Picsart-AI-Research/LIVE-Layerwise-Image-Vectorization: [CVPR 2022 Oral] Towards Layer-wise Image Vectorization

bookmark · Added on July 1, 2024 · 1 min read

The text discusses a new method called LIVE for generating SVG images layer by layer to fit raster images. LIVE uses closed bezier paths to learn visual concepts in a recursive manner. Installation instructions and references for the method are provided in the text.

Udacity CS344: Intro to Parallel Programming

bookmark · Added on July 1, 2024 · 2 min read

Intro to Parallel Programming is a free online course by NVIDIA and Udacity teaching parallel computing with CUDA. It's for developers, scientists, engineers, and students looking to learn about GPU programming and optimization. The course is self-paced, requires C programming knowledge, and offers approximately 21 hours of content.

CS 361: Systems Programming

bookmark · Added on July 1, 2024 · 1 min read

The Systems Programming course at UIC includes assigned readings, video lectures, labs, and quizzes scheduled throughout the week. Students can access additional resources and submit assignments through the course gradescope page. Office hours, content quizzes, discussions, and exams are held on specific days via Zoom and YouTube.

Resolving Rust Symbols

bookmark · Added on July 1, 2024 · 15 min read

Linking combines object files into an executable or shared library in Rust. The linker resolves symbols and dependencies between object files. Rust prefers static linking to create a single distributable binary with all dependencies included.

When FFI Function Calls Beat Native C

bookmark · Added on July 1, 2024 · 8 min read

David Yu performed a benchmark comparing different Foreign Function Interfaces (FFI) for function calls. LuaJIT's FFI was found to be faster than native C function calls due to efficient dynamic function call handling. Direct function calls, like those used by LuaJIT, can outperform indirect calls routed through a Procedure Linkage Table (PLT).

Cap'n Proto, FlatBuffers, and SBE

bookmark · Added on July 1, 2024 · 14 min read

FlatBuffers is a new serialization protocol released by Google engineers, similar to Cap’n Proto. Cap’n Proto allows random access using pointers, while FlatBuffers uses offsets stored in tables for random access. Protobufs, Cap’n Proto, and FlatBuffers have custom schema languages and different features for data serialization and access.

A Database Without Dynamic Memory Allocation

bookmark · Added on July 1, 2024 · 5 min read

TigerBeetle, a database written in Zig, does not allocate memory dynamically after startup. It uses static memory allocation for all data structures, avoiding performance issues and use-after-free bugs. This approach allows for better predictability, easier handling of overload, and efficient resource management.

Wizard Zines Collection!

bookmark · Added on July 1, 2024 · 1 min read

Julia offers programming zines with black and white covers for free and colored covers for purchase. The zines can be bought individually for $10-$12 each or as a whole collection. Additionally, there are free posters and a weekly comic subscription available.

Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0

bookmark · Added on July 1, 2024 · 15 min read

Apache Arrow DataFusion version 28.0.0 now offers faster parallel aggregation for queries with many groups. The improvements aim to enhance user experiences by generating insights more efficiently. These enhancements bring DataFusion closer to the grouping speed of DuckDB.

Problems of C, and how Zig addresses them

bookmark · Added on July 1, 2024 · 13 min read

This blog post discusses issues with C and how Zig addresses them through features like comptime evaluations and improved memory management. Zig offers solutions like error handling improvements and treating everything as an expression, making it a modern alternative to C with enhanced functionalities. The comparison highlights Zig's advantages in areas such as memory management, error handling, and expressive coding practices.

How to use hash map contexts to save memory when doing a string table

bookmark · Added on June 30, 2024 · 2 min read

The text explains how to save memory when building a string table using hash map contexts. By adapting context APIs, only indexes are stored in the table, reducing memory usage. This method can save 117 KB of memory for a string table with 10 thousand entries.

resume.txt

bookmark · Added on June 30, 2024 · 9 min read

Andrew Kelley is a programmer with 16 years of experience in software development and a passion for open-source projects. He has worked on various music-related software like the Genesis DAW and libgroove, contributing patches to libav and ffmpeg. Additionally, he has experience in low-level systems, custom algorithm creation, and designing user interfaces.

Leslie Lamport

bookmark · Added on June 28, 2024 · 3h 51m read

Leslie Lamport wrote several papers on verifying and specifying concurrent systems using TLA. He discovered algorithms through formal derivation and emphasized mechanical verification of concurrent algorithms. His work influenced the development of the TLAPS proof system.

Indices and tables

bookmark · Added on June 27, 2024 · 1 min read

CompilerGym is a library for reinforcement learning in compiler tasks. It helps ML researchers work on optimization problems and allows system developers to create new tasks for ML research. The goal is to use ML to make compilers faster.

448997590_1496256481254967_2304975057370160015_n

bookmark · Added on June 27, 2024 · 1h 0m read

The LLM Compiler is a suite of pre-trained models designed for code optimization tasks, based on Code Llama. It has been trained on a large corpus of LLVM-IR and assembly code to enhance compiler behavior understanding. The release of LLM Compiler aims to support further research in compiler optimization for both academia and industry.

Bare Bones

bookmark · Added on June 25, 2024 · 27 min read

This text explains how to create an operating system by first cross-compiling and using existing technology. It guides you through writing a kernel in C or C++, creating a bootloader, and linking the kernel for x86 systems. Following these steps ensures your operating system can be loaded and executed correctly.

The Graphics Codex

bookmark · Added on June 24, 2024 · 1 min read

"The Graphics Codex" is a comprehensive resource for computer graphics, offering essential information on 3D rendering and shading. It includes equations, diagrams, and programming projects, with free updates every month. Written by expert Morgan McGuire, it is a valuable tool for learning and reference in the field of computer graphics.

[2305.13009] Textually Pretrained Speech Language Models

bookmark · Added on June 24, 2024 · 1 min read

Notes on partial borrows

bookmark · Added on June 24, 2024 · 6 min read

The text discusses limitations of the Rust borrow checker and proposes solutions for creating references that borrow from specific subsets of a type. Two approaches, "View types" and "Reference views," are explored to address these limitations and provide more flexibility in borrowing subsets of fields with different lifetimes and mutability. The discussion includes examples, subtyping implications, monomorphization considerations, and the need to update Rust's aliasing model to accommodate view references accessing discontiguous memory regions.

Dioxus Labs + “High-level Rust”

bookmark · Added on June 24, 2024 · 23 min read

An article criticized Rust's gamedev hype, but its popularity stems from meeting modern programming needs like speed and safety. Efforts are underway to enhance Rust's capabilities for various industries and improve compile times significantly. Proposed enhancements include incremental linking, parallel frontend, and macro expansion caching to make Rust more efficient for developers.

Compile-Time Configuration For Zig Libraries

bookmark · Added on June 24, 2024 · 2 min read

To expose compile-time configuration options in Zig libraries, developers can use global declarations in the root source file or through Zig's build system. By setting configuration flags, developers can customize behavior such as enabling or disabling assertions in library code. Compile-time configuration can improve performance by allowing certain checks to be done at compile-time rather than runtime.

Generics

bookmark · Added on June 24, 2024 · 5 min read

Generics in Zig allow for creating data structures and algorithms that can work with different types. By using generics, code can be written once and reused with various data types. Zig's approach to generics involves leveraging compile-time metaprogramming capabilities.

Zig's HashMap - Part 1

bookmark · Added on June 24, 2024 · 12 min read

Zig's std.HashMap implementation relies on two key functions: hash and eql. The documentation outlines various hash map types and their functionalities, including std.HashMapUnmanaged. AutoHashMap can automatically generate hash functions, but there are limitations, and custom contexts can be provided for more complex keys.

Zig Parser

bookmark · Added on June 24, 2024 · 15 min read

The Zig Parser is a crucial part of the Zig compiler internals, responsible for constructing an abstract syntax tree from a stream of tokens. The parser uses a struct called Parser to manage the internal state of the parse operation, accumulating errors and building up AST nodes. Understanding the structure of an AST node and the data pattern is essential for comprehending how the parser works and the subsequent stages of the compiler. The AST node data is stored in various locations such as the token stream, the node list, and the extra data list, with specific structures and indexes used to access information about AST nodes like function declarations and prototypes.

Copying Better: How To Acquire The Tacit Knowledge of Experts

bookmark · Added on June 24, 2024 · 29 min read

The text discusses how to acquire expert intuition, known as tacit knowledge, through emulation and apprenticeship. Naturalistic Decision Making (NDM) research helps extract and teach expert judgment using methods like Cognitive Task Analysis and the recognition-primed decision making model. Experts rely on implicit memory and pattern recognition to make rapid assessments and decisions, which can be challenging to verbalize.

Causal ordering

bookmark · Added on June 24, 2024 · 4 min read

Causal ordering is essential for understanding distributed systems, where events may not have a clear time order. This concept helps determine the causal relationship between events in a system. It enables reasoning about causality, leading to simpler solutions in distributed computing.

Assorted thoughts on zig (and rust)

bookmark · Added on June 24, 2024 · 24 min read

Zig is simpler than Rust and offers similar features through compile-time execution. Rust provides strong type safety guarantees for generic functions, while Zig lacks automatic type constraint documentation and may face challenges with IDE support. Zig excels in custom allocators and handling out-of-memory errors, while Rust excels in preventing memory leaks and resource management.

Columnar kernels in go?

bookmark · Added on June 24, 2024 · 11 min read

Over the winter I'm going to be adding a columnar query engine to an existing system written in go.

An opinionated map of incremental and streaming systems

bookmark · Added on June 24, 2024 · 7 min read

The text discusses various design choices and characteristics of incremental and streaming systems. It highlights the core idea of these systems, which is to process inputs to generate outputs efficiently. The systems are categorized based on unstructured vs structured design, high temporal locality vs low temporal locality workloads, internal consistency vs internal inconsistency, and eager vs lazy computation approaches. The text explains the advantages and disadvantages of each design choice and provides examples of systems that fall into different categories. Additionally, it emphasizes the importance of understanding these design choices in selecting the appropriate system for specific workloads.

Internal consistency in streaming systems

bookmark · Added on June 24, 2024 · 32 min read

The text discusses the importance of internal consistency in streaming systems. It explains how eventual consistency can lead to incorrect outputs and the need for systems to wait for all relevant inputs before emitting results. Maintaining internal consistency ensures correct outputs and prevents confusion between changes and corrections.

Pain we forgot

bookmark · Added on June 24, 2024 · 15 min read

The text discusses the challenges in programming and the need for more user-friendly tools. It emphasizes the importance of improving feedback loops, running code smoothly, and creating more helpful programming environments. The author suggests rethinking traditional tools and approaches to make programming more accessible and efficient.

Have you tried rubbing a database on it?

bookmark · Added on June 24, 2024 · 1 min read

HYTRADBOI was a conference featuring lightning talks on innovative uses of databases for solving problems. Talks included topics like building data-centric apps, realtime machine learning, and interactive databases. The event focused on embracing new solutions and fostering professional behavior among attendees.

The shape of data

bookmark · Added on June 24, 2024 · 29 min read

The text discusses the importance of having a clear and consistent data notation in programming languages like Clojure. It emphasizes the advantages of a notation that closely aligns with the in-memory representation of data, making it easier for developers to work with and understand data structures. Additionally, it suggests that a well-designed data model and notation are crucial for efficient data manipulation and code analysis.

Reflections on a decade of coding

bookmark · Added on June 24, 2024 · 2 min read

The author reflects on 12 years of coding experience, sharing recent projects and personal growth insights. They highlight the importance of gradual improvements in habits and processes over innate talent. The author identifies areas of progress, like writing efficient code and managing emotions, while acknowledging gaps in experience in maintaining large codebases and teamwork.

Prospecting for Hash Functions

bookmark · Added on June 24, 2024 · 11 min read

The text discusses the process of designing non-cryptographic integer hash functions, exploring different operations and constraints to create effective hash functions. It also compares various 32-bit hash functions and their bias levels, highlighting the search for high-quality hash functions with minimal bias for both 32-bit and 64-bit integers.

The Missing Zig Polymorphism / Runtime Dispatch Reference

bookmark · Added on June 24, 2024 · 6 min read

The text discusses how Zig lacks built-in polymorphism features like interfaces or virtual methods. It explores creating polymorphism using existing language features in Zig. The author provides a detailed guide on implementing polymorphism in Zig, focusing on dynamic dispatch using function pointers.

Nanosystems

bookmark · Added on June 23, 2024 · 1 min read

This text is about a book called "Nanosystems" by K. Eric Drexler, which is considered groundbreaking in the field of molecular nanotechnology. The book explains how to create manufacturing systems at the molecular level and discusses the significant impact nanotechnology will have on various industries. Experts praise the book for providing a foundation for future research in molecular systems engineering and molecular manufacturing.

How To Become A Hacker

bookmark · Added on June 23, 2024 · 46 min read

The text explains what it means to be a hacker, focusing on problem-solving, creativity, and a willingness to share knowledge within the hacker culture. It emphasizes the importance of developing a hacker mindset, skills, and dedication through self-education and a passion for solving new problems. The hacker culture values intelligence, hard work, and a sense of community, with an emphasis on learning and sharing information to advance the collective knowledge of hackers.

the rr debugging experience

bookmark · Added on June 20, 2024 · 7 min read

rr is a debugging tool for Linux that records failures for deterministic replay under gdb. It helps debug real applications efficiently and supports reverse execution for finding bugs. rr aims to make debugging easier with low overhead and powerful features like hardware data watchpoints.

Text Buffer Reimplementation

bookmark · Added on June 19, 2024 · 15 min read

The Visual Studio Code 1.21 release includes a new text buffer implementation that improves performance in terms of speed and memory usage. The previous implementation used an array of lines, but it had limitations such as high memory usage and slow file opening times. The new implementation uses a piece table data structure, which allows for better memory usage and faster line look-up. Additionally, the implementation uses techniques such as caching for faster line lookup and a balanced binary tree for efficient searching. Benchmarks showed that the new implementation outperformed the previous line array implementation in terms of memory usage, file opening times, and reading operations.

What Is The Minimal Set Of Optimizations Needed For Zero-Cost Abstraction?

bookmark · Added on June 19, 2024 · 6 min read

Rust and C++ offer "zero-cost abstractions" where high-level code compiles to low-level code without added runtime overhead, but enabling necessary compiler optimizations can slow down compilation and impact debugging. The challenge is to find the minimal set of optimizations that maintain zero-cost abstractions while improving build speed and debug information quality. Balancing fast debuggable builds with zero-cost abstractions is crucial for performance and developer experience in languages like Rust and C++.

Using ASCII waveforms to test hardware designs

bookmark · Added on June 19, 2024 · 5 min read

Using expect tests automates the validation of code output, detecting errors efficiently. Jane Street uses Hardcaml in OCaml for hardware development, simplifying testbench creation. Waveform expect tests help visualize hardware behavior, improving development workflows.

Rust 2019 and beyond: limits to (some) growth.

bookmark · Added on June 19, 2024 · 10 min read

The text discusses the need for controls and policies to manage the growth limits of technical artifacts and the strains on individuals in the Rust project. It emphasizes the importance of acknowledging and addressing these limits to prevent potential crises or dysfunction in the future. The author suggests implementing controls, such as hard limits and moderation strategies, to maintain a healthy and sustainable project environment.

Your ABI is Probably Wrong

bookmark · Added on June 19, 2024 · 4 min read

The text discusses how most ABIs have a design flaw that harms performance by passing large structures inefficiently. Different ABIs handle passing large structures differently, but they all repeat the same mistakes. A correctly-specified ABI should pass large structures by immutable reference to avoid unnecessary copies.

GitHub - sirupsen/napkin-math: Techniques and numbers for estimating system's performance from first-principles

bookmark · Added on June 19, 2024 · 8 min read

The project "Napkin Math" aims to provide resources and techniques to estimate system performance quickly and accurately. It includes examples like estimating memory reading speed and storage costs for applications. The best way to learn this skill is through practical application, with the option to subscribe for regular practice problems. Detailed numbers and cost estimates are provided, along with compression ratios and techniques to simplify calculations. The project encourages user participation to enhance and refine the provided data and tools for napkin math calculations.

Don't write bugs

bookmark · Added on June 19, 2024 · 1 min read

Effective programmers should focus on preventing bugs rather than debugging them. Re-reading code frequently can help reduce the number of errors. Writing bug-free code is achievable with practice and attention to detail.

technicalities: "not rocket science" (the story of monotone and bors)

bookmark · Added on June 19, 2024 · 5 min read

The text discusses the development of a program called bors that enforces the "Not Rocket Science Rule" of maintaining a code repository that always passes tests. Bors automates integration testing and ensures code changes are only merged if they pass tests, preventing broken code from being merged. This system has been found to be extremely beneficial for software projects, ensuring a stable and reliable codebase.

Why is Python slow

bookmark · Added on June 19, 2024 · 5 min read

Python's performance issues stem from spending most time in the C runtime, rather than the Python code itself. Pyston focuses on speeding up the C code to improve performance. Suggestions to improve Python's speed by using other JIT techniques overlook the fundamental issue of optimizing C code.

Design duality and the expression problem

bookmark · Added on June 19, 2024 · 13 min read

The text discusses the concept of design duality in programming, focusing on the trade-offs between objects and data representations. It highlights the importance of making conscious design choices when introducing new types, whether as data, objects with extensible implementations, or abstract data types with restricted extensibility. The author emphasizes the need for programming languages to better support and encourage these design considerations.

Random Thoughts On Rust: crates.io And IDEs

bookmark · Added on June 19, 2024 · 4 min read

The author shares experiences with Rust, praising cargo and crates.io for easy code distribution. They highlight the need for improved library discovery on crates.io and discuss the potential for better IDE support in Rust projects. Despite challenges like type inference, Rust's design enables advanced IDE features that can enhance coding efficiency.

John Carmack on Inlined Code

bookmark · Added on June 19, 2024 · 13 min read

Consider inlining functions that are only called in one place for efficiency. Simplify code structure to reduce bugs and improve performance. Emphasize consistent execution paths over avoiding minor optimizations.

A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World

bookmark · Added on June 19, 2024 · 34 min read

The text discusses the development and commercialization of a bug-finding tool that can identify errors in large amounts of code. It highlights the challenges faced in finding and addressing various types of bugs, such as memory corruption and data races, across different programming systems. The tool's effectiveness in uncovering bugs in complex codebases emphasizes the importance of bug detection for improving software quality.

What is Systems Programming, Really?

bookmark · Added on June 19, 2024 · 12 min read

The term "systems programming" combines low-level programming and systems design. It involves creating and managing complex components, often focusing on machine implementation details. Over time, the distinction between systems programming and other programming languages has become less clear.

MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding

bookmark · Added on June 18, 2024 · 1 min read

MLKV introduces Multi-Layer Key-Value sharing to reduce memory usage in transformer decoding. This approach improves efficiency without sacrificing performance on NLP benchmarks. MLKV significantly reduces memory requirements compared to existing methods like Multi-Query Attention.

Mitchell Hashimoto

bookmark · Added on June 17, 2024 · 1 min read

Mitchell Hashimoto is an advisor at Polar and shares insights on technical projects, Zig programming, and automation on his website. He discusses various topics like GitHub pull requests, Zig build system, and AI growth through cloud lens. Mitchell's writing covers a range of technical subjects and his experiences in the startup world.

Understanding_Machine_Learning_-_From_Theory_to_Algorithms

bookmark · Added on June 17, 2024 · 12h 46m read

I'm sorry, but there is no content provided for me to summarize. If you provide me with the specific content or information you would like summarized, I would be happy to help.

UB Might Be a Wrong Term for Newer Languages Apr 2, 2023

bookmark · Added on June 17, 2024 · 1 min read

The author suggests that using the term "undefined behavior" in newer languages like Zig and Rust may not be the best choice due to differences in semantics. In C, implementations can define some behaviors left undefined by the standard, but in Rust and Zig, any program showing undefined behavior is considered invalid. The author proposes using terms like "non-trapping programming error" or "invalid behavior" to better convey the intended semantics in these languages.

What Every C Programmer Should Know About Undefined Behavior #1/3

bookmark · Added on June 17, 2024 · 9 min read

This blog post explains that many seemingly reasonable things in C actually have undefined behavior, leading to common bugs in programs. Undefined behavior in C allows for optimizations that improve performance but can result in unexpected outcomes like formatting your hard drive. Understanding undefined behavior is crucial for C programmers to prevent potential issues and improve code efficiency.

The Rustonomicon

bookmark · Added on June 16, 2024 · 2 min read

The Rustonomicon is a book for understanding Unsafe Rust programming details. It complements The Rust Programming Language by delving into combining language pieces and potential issues. The book covers topics like (un)safety, creating safe abstractions with unsafe primitives, and working with memory, but does not provide exhaustive API details.

chrono-Compatible Low-Level Date Algorithms

bookmark · Added on June 15, 2024 · 43 min read

The text explains algorithms for handling dates and determining leap years. It includes functions for calculating the last day of a month and converting dates between different calendar systems. The algorithms are designed to be efficient and accurate for various date calculations.

Step-by-Step Diffusion: An Elementary Tutorial

bookmark · Added on June 15, 2024 · 1h 26m read

The text is a tutorial about diffusion. The authors are Preetum Nakkiran, Arwen Bradley, Hattie Zhou, and Madhu Advani. The tutorial is available on the domain readwise.io.

So Many New Systems Programming Languages II

bookmark · Added on June 14, 2024 · 12 min read

The text discusses new systems programming languages like Rust, Zig, and Odin, highlighting their safety features and syntax. These languages offer improved memory management and safety compared to older languages like C and C++. Rust, in particular, stands out for its memory safety, threading support, and borrow checker.

zackoverflow

bookmark · Added on June 14, 2024 · 1 min read

Zack, the author, enjoys building things and delving into the inner workings of systems and computers for dopamine. He works on the Bun JavaScript runtime and creates music when not coding. Zack invites anyone to chat through his open calendar link.

From Theory To Implementation

bookmark · Added on June 14, 2024 · 1 min read

Physically Based Rendering is a widely-used textbook in computer graphics that combines theory with practical implementation for creating realistic images. The book, authored by industry experts, offers cutting-edge algorithms and ideas, including GPU ray tracing, to help readers design advanced rendering systems. Both the third and fourth editions of the book are available online for free.

Speech-to-text models

bookmark · Added on June 13, 2024 · 17 min read

Speech-to-text AI enhances communication and accessibility by transcribing spoken words into text accurately and efficiently. Machine learning and AI advancements have significantly improved the accuracy and adaptability of speech-to-text systems. These technologies open up new possibilities for inclusive and effective communication across various industries.

Ray Tracing in One Weekend

bookmark · Added on June 13, 2024 · 1h 3m read

"Ray Tracing in One Weekend" introduces readers to the concept of ray tracing through a step-by-step guide to creating a ray tracer that produces images. The document covers topics such as sending rays into the scene, ray-sphere intersection, shading, and reflection. It explains the mathematical aspects behind ray tracing, including formulas for sphere intersections and normal vectors. The guide progresses from creating a simple image of a sphere to more complex scenes, providing insights into the coding process and considerations for optimizing the rendering process.

Untangling Lifetimes: The Arena Allocator

bookmark · Added on June 13, 2024 · 40 min read

The text discusses the arena allocator as an alternative to traditional manual memory management in C, addressing issues with malloc and free. The arena allocator simplifies memory allocation and deallocation by grouping lifetimes together in a single block of memory. It provides a more efficient and manageable way to handle memory usage in codebases compared to the malloc and free approach.

Tree-Structured Concurrency — 2023-07-01

bookmark · Added on June 12, 2024 · 15 min read

Structured concurrency is a programming concept that ensures clear control flow in concurrent programs. In the context of async Rust, it guarantees properties like cancellation propagation, which means that dropping a future will also cancel all nested futures. The text discusses examples of unstructured and structured concurrency patterns, emphasizing the importance of applying structured concurrency to improve program correctness and maintainability. It also mentions the need for more API support to fully achieve structured concurrency in async Rust, suggesting practical approaches like using task queues or adopting the smol model for task spawning. Overall, structured concurrency provides a way to reason about async Rust programs effectively and enhance their reliability.

immersivemath: Immersive Linear Algebra

bookmark · Added on June 12, 2024 · 1 min read

This text introduces a book on linear algebra with chapters covering vectors, dot products, matrix operations, and more. It aims to help readers understand fundamental concepts and tools in linear algebra through clear explanations and examples. The book includes topics such as Gaussian elimination, determinants, rank, and eigenvalues.

BSTJ 57: 6. July-August 1978: The UNIX Time-Sharing System. (Ritchie, D.M.; Thompson, K.)

bookmark · Added on June 12, 2024 · 4 min read

The UNIX Time-Sharing System is a versatile operating system with unique features. It runs on Digital Equipment Corporation computers and emphasizes simplicity and ease of use. UNIX has been widely adopted for research, education, and document preparation purposes.

Principles of compiler design

bookmark · Added on June 12, 2024 · 1 min read

This text is about a book on compiler design principles. The book is authored by Jeffrey D. Ullman and contains 604 pages. It includes bibliographical references, but access to the EPUB and PDF versions is not available.

A Mathematical Theory of Communication

bookmark · Added on June 12, 2024 · 2h 24m read

The paper extends communication theory by considering noise in the channel, savings from message structure, and channel capacity. It discusses entropy, coding efficiency, channel capacity, noisy channels, equivocation, and optimal information transmission techniques. Examples and theorems are provided to explain the concepts of encoding, channel capacity, and noise in communication systems.

Mapping the whole internet with Hilbert curves

bookmark · Added on June 12, 2024 · 4 min read

The author mapped the internet using Hilbert curves to visualize IP addresses. The curves help display the vast network structure in a more comprehensible way. The scan revealed interesting patterns and changes in IP address allocations over time.

Hausdorff dimension - Wikipedia

bookmark · Added on June 12, 2024 · 1 min read

xorvoid

bookmark · Added on June 12, 2024 · 3 min read

Anthony Bonkoski, a computer enthusiast and engineer, shares his experiences in programming and working in quantitative finance. He enjoys working on various projects and has expertise in low-level programming, distributed systems, and reverse-engineering. Currently taking a break from full-time work, he is open to part-time consulting projects and enjoys writing and exploring new interests.

A Recipe for Training Neural Networks

bookmark · Added on June 11, 2024 · 19 min read

The text discusses common mistakes in training neural networks and emphasizes the importance of patience and attention to detail for successful deep learning. It provides a recipe for training neural networks, including steps like setting up a training skeleton, visualizing losses, and focusing on regularization and tuning to improve model performance. The text also highlights the value of adding more real data and using ensembles to enhance accuracy.

You own your data, in spite of the cloud

bookmark · Added on June 11, 2024 · 54 min read

The text discusses the benefits of local-first software, emphasizing ownership and control of data while also enabling seamless collaboration. It compares traditional cloud apps with new approaches that prioritize user ownership and real-time collaboration. The focus is on developing software that combines the convenience of cloud apps with the data ownership of traditional software.

Writing CUDA Kernels for PyTorch

bookmark · Added on June 11, 2024 · 34 min read

The text shows the thread distribution on different streaming multiprocessors (SM) in CUDA. Threads are organized into warps, lanes, and specific thread numbers within each SM. This information is crucial for optimizing CUDA kernels in PyTorch.

Multi-Query & Grouped-Query Attention

bookmark · Added on June 11, 2024 · 13 min read

The text explains how Multi-Query Attention and Grouped-Query Attention reduce the Key-Value Cache size in transformer models while maintaining performance. Multi-Query Attention allows multiple attention heads to share key and value vectors, while Grouped-Query Attention groups these vectors based on a hyperparameter, offering a balance between performance and cache reduction. These techniques help manage memory usage during text generation tasks in transformer models.

999 crates of Rust on the wall

bookmark · Added on June 11, 2024 · 9 min read

The author compared popular crates on crates.io to their upstream repositories to improve supply chain security. Most top crates matched their repositories, but some had issues like missing VCS info or build failures. Future work includes extending this analysis to all crates on crates.io and improving publishing processes for better security.

Uiuisms

bookmark · Added on June 9, 2024 · 1 min read

This text provides a list of Uiua functions for solving common problems. Contributors can add more functions to the list in the repository. Functions include splitting arrays, removing rows, upscaling matrices, and working with diagonal arrays.

Arithmetic functions

bookmark · Added on June 9, 2024 · 7 min read

BQN's arithmetic functions mirror mathematical notation and apply element-wise to arrays. BQN supports basic arithmetic operations like addition, subtraction, multiplication, division, exponentiation, and root functions. Character arithmetic is a distinctive feature allowing manipulation of characters with symbols like + and -.

An interactive study of queueing strategies

bookmark · Added on June 9, 2024 · 11 min read

This text explores different queueing strategies for handling requests, emphasizing the importance of prioritizing requests effectively to prevent dropping important ones. It introduces concepts like FIFO and priority queues, as well as active queue management techniques to optimize request processing. Understanding these strategies can help in efficiently managing queues and improving overall system performance.

A DSL for Implementing Math Functions

bookmark · Added on June 8, 2024 · 1 min read

ethereumbook/04keys-addresses.asciidoc at develop · ethereumbook/ethereumbook · GitHub

bookmark · Added on June 6, 2024 · 34 min read

This chapter introduces public key cryptography used in Ethereum for securing ownership of funds through private keys and addresses. Public keys are derived from private keys and are represented as points on an elliptic curve. Ethereum addresses are unique identifiers generated from public keys using the Keccak-256 hash function.

Accidentally Turing-Complete

bookmark · Added on June 6, 2024 · 8 min read

The document "Accidentally Turing-Complete" explores various unexpected systems and technologies that unintentionally exhibit Turing completeness, a property that allows them to perform any computation. Examples include C++ templates, TypeScript, Java generics, X86 mov instructions, Magic: The Gathering card game, HTML5, Minecraft, Dwarf Fortress game, SQL, Apache Rewrite Rules, Pokemon Yellow game, Scala type system, MediaWiki templates, Little Big Planet game, Sendmail, Vim Normal-Mode, Border Gateway Protocol (BGP), Excel, Super Mario World glitches, PowerPoint, Font Shaping, JBIG2 Image Compression, and Stupid RDMA NICs. The document showcases how these diverse systems, from games to internet protocols, can unexpectedly demonstrate the computational power of Turing completeness.

The Art of Computer Programming, Vol. 4 Fascicle 6

bookmark · Added on June 6, 2024 · 16h 13m read

The_Manga_Guide_to_Linear_Algebra

bookmark · Added on June 6, 2024 · 1h 47m read

Exploring architectures- Transformers II

bookmark · Added on June 6, 2024 · 6 min read

The text explains how Transformers utilize queries, keys, and values to calculate self-attention weights for tokens. It details the process of obtaining the self-attention weights and generating output tokens through neural networks. The final steps involve calculating loss using cross-entropy and backpropagating to update the weight parameters.

What are Diffusion Models?

bookmark · Added on June 6, 2024 · 23 min read

Diffusion models slowly add noise to data and then learn to reverse the process to create desired samples. Unlike other models, diffusion models have a fixed procedure and high-dimensional latent variables. Training a diffusion model involves approximating conditioned probability distributions and simplifying the objective function.

Problems with BQN

bookmark · Added on June 6, 2024 · 30 min read

BQN has issues with incoherent monad-dyad pairs and train structures, making code readability and implementation challenging. Modifications like the Constant modifier ˙ attempt to address these challenges. However, there are still limitations in tacit code construction and array reductions that impact the language's usability.

Iterative α-(de)Blending: a Minimalist Deterministic Diffusion Model

bookmark · Added on June 5, 2024 · 1 min read

The paper presents a simple and effective denoising-diffusion model called Iterative α-(de)Blending. It offers a user-friendly alternative to complex theories, making it accessible with basic calculus and probability knowledge. By iteratively blending and deblending samples, the model converges to a deterministic mapping, showing promising results in computer graphics applications.

The borrow checker within

bookmark · Added on June 5, 2024 · 15 min read

The text discusses improvements to Rust's borrow checker to align better with its core design ethos of mutation xor sharing. These changes aim to make Rust code patterns feel more intuitive and work seamlessly with the borrow checker's rules. The proposed enhancements include features like conditional return references, view types, and addressing phased initialization issues.

How should I read type system notation?

bookmark · Added on June 5, 2024 · 11 min read

A type system in programming languages follows rules for expressions and types. Typing rules are written as relationships between expressions and their types for checking and inferring types. Contexts are used to keep track of variable types in type judgments.

Writing a Simple Garbage Collector in C

bookmark · Added on June 5, 2024 · 19 min read

Summary: The text explains how to implement a simple garbage collector in C by writing a memory allocator function that manages free and used memory blocks using linked lists. The garbage collection algorithm involves scanning memory regions to mark blocks in use and free those not in use. The collector function collects unused memory blocks, making the heap scanning code simpler and faster.

A decade of developing a programming language

bookmark · Added on June 5, 2024 · 14 min read

The author spent a decade developing the programming language Inko, transitioning from gradual to static typing and using Rust for the compiler. Recommendations include avoiding gradual typing, self-hosting compilers, and focusing on functionality over performance when building a new language. Building a language for long-term use is a time-consuming process that requires prioritizing user needs over technical complexities.

The Rust I Wanted Had No Future

bookmark · Added on June 5, 2024 · 15 min read

The author preferred certain design choices in early Rust over the current state, such as the treatment of certain language features and performance considerations. They express a desire for a simpler, less performance-focused language with different priorities than those commonly held in the Rust community. The author reflects on their preferences for language design and the trade-offs they would have made for a more straightforward and expressive programming experience.

The Garbage Collection Handbook

bookmark · Added on June 5, 2024 · 2 min read

The Garbage Collection Handbook is a comprehensive guide on automatic memory management, covering modern techniques and challenges faced by programmers. This second edition updates the handbook with insights from over 60 years of research and development in the field. It is essential reading for programmers looking to understand and navigate the complexities of garbage collection in modern programming languages.

A high-bias, low-variance introduction to Machine Learning for physicists

bookmark · Added on June 5, 2024 · 6h 46m read

This text is an introduction to Machine Learning for physicists, highlighting the natural connections between ML and statistical physics. It explains the use of "energy-based models" inspired by statistical physics in deep learning methods. The discussion includes the application of methods from statistical physics to study deep learning and the efficiency of learning rules.

How diffusion models work: the math from scratch

bookmark · Added on June 1, 2024 · 18 min read

Diffusion models generate diverse high-resolution images and are different from previous generative methods. Cascade diffusion models and latent diffusion models are used to scale up models to higher resolutions efficiently. Score-based generative models are similar to diffusion models and involve noise perturbations to generate new samples.

essentials-of-compilation

bookmark · Added on May 31, 2024 · 2h 54m read

The text discusses the implementation of compilers for different programming languages, covering topics such as syntax definitions, interpreter extensions, and x86 assembly translation. It emphasizes simplifying the compiler process for readers by using a straightforward language and providing step-by-step guidance on compiler development. Additionally, it introduces new language features like Booleans, conditionals, and tuples, expanding the capabilities of the compilers being built.

PRACTICAL COMPILER CONSTRUCTION

bookmark · Added on May 31, 2024 · 1 min read

"Practical Compiler Construction" is a textbook on writing compilers with annotated source code. The second edition is now available in print with improvements and bug fixes. The book covers compiler construction concepts and advanced techniques for optimizing code.

A Distributed Systems Reading List

bookmark · Added on May 31, 2024 · 4 min read

This reading list covers materials for understanding distributed systems design and challenges. It includes resources on topics like latency, Amazon's organizational culture, Google's cutting-edge technologies, consistency models, theory, languages, tools, infrastructure, storage, Paxos consensus, and gossip protocols. The list aims to help readers adapt their thinking to effectively tackle distributed system complexities.

An Introduction to Assembly Programming with RISC-V

bookmark · Added on May 28, 2024 · 26 min read

This text provides information about a resource related to RISC-V programming. The ISBN number for this resource is 978-65-00-15811-3. It is authored by riscv-programming.org.

Microsoft PowerPoint - SRAM Architecture

bookmark · Added on May 28, 2024 · 4 min read

The text discusses the architecture of Static Random Access Memory (SRAM) cells, focusing on their read and write operations, sizing considerations, and column circuitry. SRAM cells store data using cross-coupled inverters, with specific steps for reading and writing data. Column circuitry includes bitline conditioning, sense amplifiers, and multiplexing for efficient data access.

MLIR: A Compiler Infrastructure for the End of Moore's Law

bookmark · Added on May 27, 2024 · 59 min read

MLIR is a versatile compiler infrastructure designed to address software fragmentation and improve compilation for different hardware. It aims to reduce the cost of building domain-specific compilers and facilitate the connection of existing compilers. MLIR offers a standardized approach to code generation and optimization across various application domains and hardware targets.

MLIR — Getting Started

bookmark · Added on May 27, 2024 · 13 min read

The text is a guide titled "MLIR — Getting Started" by Math ∩ Programming available on www.jeremykun.com.

Chapter 2 Basics of SIMD Programming

bookmark · Added on May 27, 2024 · 45 min read

The text explains how to organize data for SIMD operations and provides examples of SIMD-Ready Vectors. It also discusses the relationship between vectors and scalars in SIMD programming. Built-in functions for VMX instructions and SIMD operation principles are outlined in the text.

Matrix multiplication in Mojo

bookmark · Added on May 27, 2024 · 10 min read

The text discusses matrix multiplication in Mojo. It is written by modular.com and can be found on docs.modular.com.

Matrix Multiplication on CPU

bookmark · Added on May 27, 2024 · 31 min read

The text is about matrix multiplication on a CPU. The author is Marek Kolodziej and the domain is marek.ai.

How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog

bookmark · Added on May 27, 2024 · 31 min read

The text is a worklog by Simon Boehm about optimizing a CUDA Matmul Kernel for cuBLAS-like performance. It can be found on the domain siboehm.com.

The Annotated Transformer

bookmark · Added on May 27, 2024 · 32 min read

The text discusses the architecture and training of a Transformer model. It explains the use of self-attention and feed-forward networks in the encoder and decoder. The model is demonstrated through examples of prediction and visualization of attention mechanisms.

Anonymity and the internet

bookmark · Added on May 27, 2024 · 3 min read

Anonymity on the internet is fragile, with each piece of information reducing anonymity. Revealing multiple bits of personal information can jeopardize anonymity, but deliberate disinformation can help regain some anonymity. To protect anonymity, it's best to minimize information disclosure.

Auto-Regressive Next-Token Predictors are Universal Learners

bookmark · Added on May 26, 2024 · 40 min read

Simple linear next-token predictors can efficiently approximate any function computable by a Turing machine. Even basic models like linear networks and shallow Multi-Layer Perceptrons show strong performance on tasks like text generation and arithmetic. By leveraging auto-regressive learning, these models can achieve impressive results in solving complex tasks.

Where Vim Came From

bookmark · Added on May 25, 2024 · 14 min read

Vim is a popular text editor with a long history tracing back to the Unix epoch. Its development started in 1988 and evolved from the "wq text editor" concept. Vim's success is attributed to its features and the gradual accumulation of good ideas over time.

Building and operating a pretty big storage system called S3

bookmark · Added on May 25, 2024 · 31 min read

Dr. Werner Vogels shares insights from working on Amazon's S3 storage system, highlighting the scale and unique challenges faced. S3's design incorporates innovative strategies to efficiently handle vast amounts of data across millions of hard drives while prioritizing customer experience. Vogels emphasizes the need for a broader perspective on software systems and the rewarding journey of scaling as an engineer at Amazon.

Unnamed Document

bookmark · Added on May 25, 2024 · 1 min read

LADW_2017-09-04

bookmark · Added on May 25, 2024 · 7h 55m read

This text discusses properties of vector spaces and matrices, particularly focusing on bases and eigenvalues. It establishes that any linearly independent system of vectors can be completed to form a basis in a finite-dimensional vector space. Additionally, it explains that operators in inner product spaces have an upper triangular matrix representation under certain conditions.

New Scaling Laws for Large Language Models

bookmark · Added on May 25, 2024 · 7 min read

DeepMind's new paper challenges existing scaling laws for training large language models, proposing more optimal use of compute resources. By training a smaller 70-billion parameter model using their new scaling laws, DeepMind demonstrated superior performance compared to larger models like GPT-3 and their own 270-billion parameter model. This discovery may lead to more cost-effective and efficient training of large language models in the future.

Binary Magic: Building BitNet 1.58bit Using PyTorch from Scratch

bookmark · Added on May 25, 2024 · 5 min read

The document discusses the creation of a 1.58bit model called BitNet using PyTorch from scratch, which can rival full precision LLMs. Quantization, the process of representing float numbers with fewer bits, is explained as a method to increase the speed and reduce the RAM consumption of ML models, albeit with some loss of accuracy. BitNet differs from existing quantization approaches as it trains the model from scratch with quantization, offering a unique quantization algorithm and implementation in PyTorch. Results from experiments with custom PyTorch implementations show that the 2bit and 1bit variants of models perform as well as full precision models, demonstrating the potential of this approach.

king - man + woman is queen; but why?

bookmark · Added on May 25, 2024 · 9 min read

The text explains how the word2vec algorithm transforms words into vectors for analyzing similarities and relationships between words. By using vector arithmetic, it can find analogies such as "king - man + woman = queen." Understanding word co-occurrences can provide insight into the meaning of words through the distributional hypothesis.

How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study

bookmark · Added on May 25, 2024 · 12 min read

1-bit Model

bookmark · Added on May 25, 2024 · 7 min read

Quantizing small models like Llama2-7B at 1-bit yields poor performance but fine-tuning with low-rank adapters significantly improves output quality. The HQQ+ approach shows potential in extreme low-bit quantization for machine learning models, reducing memory and computational requirements while maintaining performance. Training larger models with extreme quantization can lead to superior performance compared to training smaller models from scratch.

Human Knowledge Compression Contest

bookmark · Added on May 25, 2024 · 48 min read

The Human Knowledge Compression Contest measures intelligence through data compression ratios. Better compression leads to better prediction and understanding, showcasing a link between compression and artificial intelligence. The contest aims to raise awareness of the relationship between compression and intelligence, encouraging the development of improved compressors.

Heatmaps and CNNs Using Fast.ai

bookmark · Added on May 25, 2024 · 8 min read

The text discusses heatmaps, CNNs, and their relationship in deep learning. It explains how heatmaps are generated using Grad-CAM heatmaps from the final layer of a Convolutional Neural Network. The article also touches on creating heatmaps using Adaptive Pooling layers and interpreting top losses for model evaluation.

Where do LLMs spend their FLOPS?

bookmark · Added on May 19, 2024 · 8 min read

LLMs (large language models) spend their FLOPS (floating point operations) on various tasks, including computing QKV (query, key, value) matrices, attention output matrices, and running the feed-forward network (FFN). The attention mechanism plays a crucial role in LLMs, even though the FLOPS required for attention calculations are relatively small. The KV cache, which stores information for each token, requires significant memory but is necessary for generating sequences. Different architectural choices, such as grouped query attention and sliding window attention, can affect the size and efficiency of the KV cache. Increasing the number of layers in an LLM linearly scales the FLOPS and parameters, while increasing the model width quadratically scales the model size. Wider models parallelize better, while deeper models increase inference time linearly.

The Annotated Diffusion Model

bookmark · Added on May 19, 2024 · 27 min read

A neural network learns to denoise data by gradually removing noise. The process involves adding noise to an image and then training the network to reverse the denoising. The network predicts noise levels based on corrupted images at different time steps.

Defusing Diffusion Models

bookmark · Added on May 19, 2024 · 21 min read

This post explains the concepts of forward and reverse diffusion processes in diffusion models. By understanding these processes, readers can train diffusion models to generate samples from target distributions effectively. Guided diffusion models are also discussed, showing how conditioning information can be used to guide the diffusion process for specific outcomes.

The Illustrated Stable Diffusion

bookmark · Added on May 19, 2024 · 11 min read

AI image generation with Stable Diffusion involves an image information creator and an image decoder. Diffusion models use noise and powerful computer vision models to generate aesthetically pleasing images. Text can be incorporated to control the type of image the model generates in the diffusion process.

Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentation

bookmark · Added on May 10, 2024 · 16 min read

Mamba-UNet is a new architecture combining U-Net with Mamba technology for better medical image segmentation performance. It addresses limitations in modeling long-range dependencies within medical images. Results show that Mamba-UNet outperforms other UNet variations in medical image segmentation tasks.

Sparse Autoencoders Find Highly Interpretable Features in Language Models

bookmark · Added on May 1, 2024 · 1 min read

Sparse autoencoders help identify clear and understandable features in language models by tackling the issue of polysemanticity. By using sparse autoencoders, researchers can pinpoint specific features responsible for certain behaviors in neural networks more effectively than other methods. This approach may lead to increased transparency and control over language models in the future.

KAN: Kolmogorov–Arnold Networks

bookmark · Added on May 1, 2024 · 1h 25m read

Kolmogorov-Arnold Networks (KANs) have learnable activation functions on edges, outperforming Multilayer Perceptrons (MLPs) in accuracy and interpretability. KANs show faster neural scaling laws than MLPs, leveraging splines and MLPs to improve accuracy and interpretability. KANs can represent functions effectively and display more favorable scaling curves than MLPs, especially in high-dimensional examples.

KAN: Kolmogorov-Arnold Networks

bookmark · Added on May 1, 2024 · 1h 43m read

KANs outperform MLPs in accuracy and interpretability by using learnable activation functions on edges. They have faster neural scaling laws and can represent special functions more efficiently. KANs offer a promising alternative to MLPs in various applications, showcasing improved performance and interpretability.

Structure and Interpretation of Computer Programs, 2nd ed.

bookmark · Added on April 30, 2024 · 15h 41m read

The text discusses key concepts in programming, such as primitive expressions, means of combination, and means of abstraction. It highlights the role of the environment in determining the meaning of symbols in expressions. The evaluation process involves reducing expressions to procedures applied to arguments, leading to a deeper understanding of programming concepts.

OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework

bookmark · Added on April 25, 2024 · 26 min read

The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and enabling investigations into data and model biases, as well as potential risks. To this end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. For example, with a parameter budget of approximately one billion parameters, OpenELM exhibits a 2.36% improvement in accuracy compared to OLMo while requiring $2\times$ fewer pre-training tokens. Diverging from prior practices that only provide model weights and inference code, and pre-train on private datasets, our release includes the complete framework for training and evaluation of the language model on publicly available datasets, including training logs, multiple checkpoints, and pre-training configurations. We also release code to convert models to MLX libra...

þÿAn Infinitely Large Napkin

bookmark · Added on April 23, 2024 · 26h 50m read

The text is titled "An Infinitely Large Napkin" by Evan Chen. The author's work can be found on readwise.io.

Transformers Meet Small Datasets

bookmark · Added on April 10, 2024 · 1 min read

Root Mean Square Layer Normalization

bookmark · Added on April 9, 2024 · 32 min read

Terry A. Davis

bookmark · Added on April 7, 2024 · 16 min read

Terry A. Davis, an American electrical engineer and programmer, created TempleOS, a public domain operating system. Despite his mental health challenges, Davis gained an online following for his unique work and beliefs. His legacy continues to be remembered through documentaries and online discussions.

Pattern Recognition and Machine Learning

bookmark · Added on April 6, 2024 · 22h 37m read

The content discusses likelihood functions for Gaussian distributions, maximizing parameters using observed data, Bayesian model comparison, mixture density networks, and EM algorithm for Gaussian mixtures. It covers topics like posterior distributions, predictive distributions, graphical models, and variational inference. The material emphasizes probability distributions, optimization, and model comparison.

Ludwig Wittgenstein: The Duty of Genius

bookmark · Added on April 6, 2024 · 20h 33m read

The text discusses the complex relationship between Ludwig Wittgenstein and his peers, particularly Bertrand Russell. Wittgenstein's philosophical ideas and personal struggles are highlighted, showing the challenges he faced in expressing his thoughts and finding understanding from others. Despite his brilliance, Wittgenstein's life was marked by loneliness and inner turmoil, making it difficult for him to fully convey his philosophical insights.

Generative Agents: Interactive Simulacra of Human Behavior

bookmark · Added on March 28, 2024 · 1h 35m read

Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks

bookmark · Added on March 11, 2024 · 5h 7m read

The text is a comprehensive survey of 400 activation functions for neural networks. It provides numerous URLs and DOIs for further reading and reference. The authors are Vladimír Kunc and Jiří Kléma.

Revisiting Deep Learning as a Non-Equilibrium Process

bookmark · Added on March 8, 2024 · 8 min read

The document discusses the nature of Deep Learning systems, highlighting differences from traditional machine learning systems and challenging common misconceptions. It emphasizes the complexity and non-convexity of Deep Learning, noting that optimization techniques alone cannot explain its success. The text critiques the field for lacking in-depth exploration of the true nature of Deep Learning, pointing out a tendency towards superficial explanations and reliance on celebrity figures rather than rigorous scientific inquiry. It delves into the use of Bayesian techniques, the role of noise, and the importance of architecture in Deep Learning, arguing for a deeper understanding of the underlying processes and the need for more precise language and theoretical exploration.

Dissipative Adaptation: The Origins of Life and Deep Learning

bookmark · Added on March 8, 2024 · 7 min read

The document explores the concept of Dissipative Adaptation, drawing parallels between the emergence of life and the mechanisms of Deep Learning. It discusses the work of Jeremy England and his theory of non-equilibrium statistical mechanics known as Dissipative Adaptation, which explains the self-organizing behavior of Deep Learning. The text delves into how neural networks evolve through training, emphasizing the role of external observations in driving the system towards minimizing entropy. It contrasts the mechanisms of Dissipative Adaptation with current Deep Learning architectures, highlighting similarities in alignment of components to maximize energy dissipation or information gradient.

A Gentle Introduction to LLVM IR

bookmark · Added on March 6, 2024 · 31 min read

Learning LLVM IR can be beneficial for generalist working programmers to understand what their compiler is doing to create highly optimized code. LLVM IR is well-documented and can be treated as a slightly weird programming language. It is strongly typed and requires explicit type annotations. LLVM IR is a static single assignment form (SSA) IR and has properties that make optimizations simpler to write. It supports control flow operations, arithmetic instructions for different types, and memory operations. There are also LLVM intrinsics available for specific functions. However, some parts of LLVM's semantics, such as undefined behavior and pointer provenance, can be challenging to navigate.

Re: [Fis] A PROPOSAL ABOUT THE DEFINITION OF INFORMATION

bookmark · Added on March 6, 2024 · 13 min read

The email exchange discusses the concept of negative entropy and its implications in mathematics and thermodynamics. Sungchul Ji questions the validity of negative entropy based on the Third Law of Thermodynamics. Arturo Tozzi argues for the existence of negative entropy in certain cases and relates it to information theory and free energy.

The Art of Embeddings: Transforming Text for Vector Databases (Part 2)

bookmark · Added on March 6, 2024 · 13 min read

Embeddings are a crucial component of transforming text into vectors in vector databases. They capture rich context and make data more useful by capturing meaning and context in a machine-readable format. Tokenization is the first step in the embedding process, where text is broken down into smaller parts or tokens. Word2Vec is a popular method that creates dense vector representations of word features based on context. However, it has limitations such as struggling with polysemy and out-of-vocabulary words. Sub-word tokenization is a hybrid approach that can handle these limitations by decomposing words into meaningful sub-words. Transformer models, such as BERT, are used to transform tokenized words into embeddings by leveraging self-attention mechanisms and positional encodings. The choice of tokenization method can significantly affect the size and effectiveness of the embeddings, including vocabulary size, handling of out-of-vocabulary words, and overall quality and usefulness of the embeddings. Choosing th...

Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

bookmark · Added on March 6, 2024 · 34 min read

The text discusses a method called Parameter-Efficient Sparsity Crafting (PESC) that enhances sparse models for natural language processing tasks. PESC involves integrating adapters into sparse models, improving performance without changing individual weights. The approach outperforms other sparse models and even competes with GPT-3.5 in various tasks.

þÿThe Little Book of Deep Learning

bookmark · Added on March 6, 2024 · 1h 36m read

I'm sorry, but there is no content provided to summarize. If you have any text or information you would like me to summarize, please provide it so I can assist you.

Information

bookmark · Added on March 6, 2024 · 14 min read

The text discusses the challenges and complexities of measuring and quantifying information, particularly in terms of storage capacity, compression, and entropy. It explores various examples, such as genome information, human sensory capabilities, and the information content of objects like water molecules and black holes. The relationship between information, entropy, and physical properties is also highlighted.

Sequence to Sequence Learning with Neural Networks

bookmark · Added on March 6, 2024 · 26 min read

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

bookmark · Added on March 6, 2024 · 16 min read

The article introduces a new era of 1-bit Large Language Models (LLMs) that can significantly reduce the cost of LLMs while maintaining their performance. BitNet b1.58 is a 1.58-bit LLM variant in which every parameter is ternary, taking on values of {-1, 0, 1}. It retains all the benefits of the original 1-bit BitNet, including its new computation paradigm, which requires almost no multiplication operations for matrix multiplication and can be highly optimized. Moreover, BitNet b1.58 offers two additional advantages: its modeling capability is stronger due to its explicit support for feature filtering, and it can match full precision (i.e., FP16) baselines in terms of both perplexity and end-task performance at a 3B size.

How to round to 2 decimals with Python? [duplicate]

bookmark · Added on March 5, 2024 · 2 min read

To round a number to 2 decimals in Python, the usual method is using round(value, significantDigit), but it can behave unexpectedly when the digit before the one being rounded is a 5. To address this, a workaround involves adding a small value to ensure proper rounding. This method allows for traditional rounding commonly used in statistics without needing to import additional libraries like Decimal. By incorporating this workaround into a function, you can achieve the desired rounding results without encountering the issue with numbers ending in 5.

Rounding floats with f-string [duplicate]

bookmark · Added on March 5, 2024 · 1 min read

Using %-formatting, I can specify the number of decimal cases in a string: x = 3.14159265 print('pi = %0.2f' %x) This would give me: pi = 3.14 Is there any way of doing this using f-strings in ...

Latent Interfaces

bookmark · Added on March 3, 2024 · 5 min read

In a career shift, the author is launching Latent Interfaces to apply expertise in design, prototyping, and development to complex data challenges. They share insights into a genomic data project, emphasizing the importance of Python skills alongside JavaScript. The document showcases the creation of intuitive data interfaces and the design process involving both digital and physical tools. Additionally, the author discusses the significance of well-designed APIs like StabilityAI and the potential for future collaborations in data visualization projects.

Hypercomputation

bookmark · Added on March 3, 2024 · 16 min read

Hypercomputation and super-Turing computation involve models of computation that can produce non-Turing-computable outputs. Introduced in the early 1990s, super-Turing computing is inspired by neurological and biological systems and serves as the foundation for Lifelong Machine Learning. Hypercomputation, a field introduced in the late 1990s, includes philosophical constructs and aims to compute functions beyond what a Turing machine can. The Church-Turing thesis states that any "computable" function can be computed by a Turing machine, but hypercomputers can compute functions that are not computable in the Church-Turing sense. Various hypercomputer models exist, ranging from theoretical concepts like oracle machines to more plausible models like quantum computing. Some proposals suggest that hypercomputation may be achievable through systems like neural networks or analog computers. Critics argue that hypercomputation is not physically realizable.

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

bookmark · Added on February 28, 2024 · 1 min read

Recent research is leading to a new era of 1-bit Large Language Models (LLMs), such as BitNet, introducing a variant called BitNet b1.58 where every parameter is ternary {-1, 0, 1}. This model matches the performance of full-precision Transformer LLMs while being more cost-effective in terms of latency, memory, throughput, and energy consumption. The 1.58-bit LLM sets a new standard for training high-performance and cost-effective models, paving the way for new computation methods and specialized hardware designed for 1-bit LLMs.

How Netflix Really Uses Java

bookmark · Added on February 27, 2024 · 39 min read

The discussion at Netflix delves into how Java is utilized within the company's architecture, highlighting their transition to Java 17 and ongoing testing with Java 21. The move to newer Java versions resulted in significant performance improvements, such as 20% better CPU usage with Java 17. Additionally, the implementation of GraphQL Federation and virtual threads in Java 21 are key advancements that are expected to impact the way code is written and scaled within Netflix's Java stack. The company's shift from Java 8 to Java 17 and the ongoing evolution of their technology frameworks and tooling, particularly focusing on Spring Boot, demonstrate their commitment to staying current with Java developments.

Scheduling Internals

bookmark · Added on February 27, 2024 · 25 min read

The document delves into the concept of concurrency in programming, exploring how tasks can be handled concurrently using different methods like threads, async I/O, event loops, and schedulers. It discusses the challenges and benefits of each approach, illustrating examples in C code to demonstrate the practical implementations. The text covers topics like preemptive and non-preemptive schedulers, implementation details in languages like Go and Rust, as well as the use of event loops for efficient task handling. It also touches on the importance of understanding program state management and the impact on task execution in concurrent programming.

Glossary of Deep Learning: Word Embedding

bookmark · Added on February 26, 2024 · 6 min read

Word embedding is a method that transforms text into numerical vectors for machine learning algorithms to process efficiently. These vectors are created to represent words or phrases as real numbers, focusing on dimensionality reduction and contextual similarity. Word2Vec is a popular algorithm that implements this approach using techniques like CBOW and Skip-gram to predict target words based on their context. While word embeddings are not deep learning themselves, they provide a way for deep nets to interpret and understand natural language, offering a new understanding of language as numbers.

gemini_v1_5_report

bookmark · Added on February 18, 2024 · 1h 59m read

Gemini 1.5 Pro is a highly compute-efficient multimodal model that can recall and reason over millions of tokens of context, including long documents, videos, and audio. It achieves near-perfect recall on long-context retrieval tasks and outperforms the state-of-the-art in long-document QA, long-video QA, and long-context ASR. Gemini 1.5 Pro also showcases surprising new capabilities, such as learning to translate a new language from a grammar manual. The model surpasses the previous Gemini 1.0 Pro and performs at a similar level to 1.0 Ultra on a wide range of benchmarks while requiring less compute to train.

How to Use t-SNE Effectively

bookmark · Added on February 16, 2024 · 9 min read

t-SNE plots can be useful for visualizing high-dimensional data, but they can also be misleading if not interpreted correctly. The technique creates 2D "maps" of data with many dimensions, but these images can be misread. The perplexity parameter, which balances attention between local and global aspects of the data, has a significant impact on the resulting plots. Different perplexity values may be needed to capture different aspects of the data. t-SNE plots can equalize cluster sizes and distort distances between clusters, making it difficult to interpret relative sizes and distances. It's important to recognize random noise and avoid misinterpreting it as meaningful patterns. t-SNE plots can show some shapes accurately, but local effects and clumping can also affect the interpretation. For topological information, multiple plots at different perplexities may be required. Overall, using t-SNE effectively requires understanding its behavior and limitations.

Consciousness, Cognition and the Neuronal Cytoskeleton – A New Paradigm Needed in Neuroscience

bookmark · Added on February 14, 2024 · 58 min read

Viewing the brain as a complex computer of simple neurons is insufficient to explain consciousness and cognition. A new paradigm is needed that considers the brain as a scale-invariant hierarchy, with quantum and classical processes occurring in cytoskeletal microtubules inside neurons. Evidence shows that microtubules regulate specific firings of axonal branches and modulate membrane and synaptic activities. This new paradigm suggests that information processing for cognitive and conscious brain functions occurs in microtubules and involves both top-down and bottom-up regulation within the brain hierarchy. The precise mechanisms of consciousness may be most likely to reveal themselves in Layer V cortical pyramidal neurons, which have a large collection of mixed polarity microtubule networks.

OpenMEA: Open-Source Microelectrode Array Platform for Bioelectronic Interfacing

bookmark · Added on February 14, 2024 · 50 min read

OpenMEA is an open-source platform for closed-loop bioelectronics research that aims to revolutionize the treatment of medical disorders and augment physiology. It includes designs for components such as electrophysiological recording and stimulation electronics, a microfluidic perfusion system, and physical designs for multielectrode arrays. The platform enables researchers to conduct in vitro experiments and understand the long-term effects of electrical stimulation and drug interactions. OpenMEA offers high-performance processing capabilities and supports simultaneous recording and stimulation, as well as the real-time adaptation of neuromodulation waveforms. It fills the gaps in existing solutions and provides a versatile tool for bioelectronic research.

Landauer's principle

bookmark · Added on February 14, 2024 · 7 min read

Landauer's principle is a physical principle that establishes the minimum energy consumption of computation. It states that irreversible changes in information stored in a computer dissipate a minimum amount of heat to the surroundings. The principle was proposed by Rolf Landauer in 1961 and states that the minimum energy needed to erase one bit of information is proportional to the temperature at which the system is operating. While the principle is widely accepted, it has faced challenges in recent years. However, it has been shown that Landauer's principle can be derived from the second law of thermodynamics and the entropy change associated with information gain.

Bremermann's limit

bookmark · Added on February 14, 2024 · 2 min read

Bremermann's limit is a maximum rate of computation that can be achieved in a self-contained system in the material universe. It is based on Einstein's mass-energy equivalency and the Heisenberg uncertainty principle. This limit has implications for designing cryptographic algorithms, as it can determine the minimum size of encryption keys needed to create an uncrackable algorithm. The limit has also been analyzed in relation to the maximum rate at which a system with energy spread can evolve into an orthogonal state.

Bekenstein bound

bookmark · Added on February 14, 2024 · 10 min read

The Bekenstein bound is an upper limit on the entropy or information that can be contained within a given finite region of space with a finite amount of energy. It implies that the information of a physical system must be finite if the region of space and energy are finite. The bound was derived from arguments involving black holes and has implications for thermodynamics and general relativity. It can be proven in the framework of quantum field theory and has applications in various fields, such as black hole thermodynamics and the study of human brains.

numerical_recipes

bookmark · Added on February 14, 2024 · 39h 52m read

The content provided is the table of contents for a book titled "Numerical Recipes: The Art of Scientific Computing, Third Edition." It includes various topics such as linear algebra, interpolation and extrapolation, integration of functions, evaluation of functions, special functions, random numbers, sorting and selection, root finding and nonlinear sets of equations, minimization or maximization of functions, eigensystems, and more.

Temperature as Joules per Bit

bookmark · Added on February 14, 2024 · 21 min read

Deep Learning Course

bookmark · Added on February 10, 2024 · 5 min read

This document provides resources for François Fleuret's deep-learning course at the University of Geneva. The course offers a thorough introduction to deep learning, with examples using the PyTorch framework. The materials include slides, recordings, and a virtual machine. The course covers topics such as machine learning objectives, tensor operations, automatic differentiation, gradient descent, and deep-learning techniques. The document also includes prerequisites for the course, such as knowledge of linear algebra, differential calculus, Python programming, and probability and statistics.

Memory in Plain Sight: A Survey of the Uncanny Resemblances between Diffusion Models and Associative Memories

bookmark · Added on February 9, 2024 · 57 min read

Diffusion Models and Associative Memories show surprising similarities in their mathematical underpinnings and goals, bridging traditional and modern AI research. This connection highlights the convergence of AI models towards memory-focused paradigms, emphasizing the importance of understanding Associative Memories in the field of computation. By exploring these parallels, researchers aim to enhance our comprehension of how models like Diffusion Models and Transformers operate in Deep Learning applications.

2309.10668

bookmark · Added on February 8, 2024 · 37 min read

This article discusses the relationship between language modeling and compression. The authors argue that large language models can be viewed as powerful compressors due to their impressive predictive capabilities. They demonstrate that these models can achieve state-of-the-art compression rates across different data modalities, such as images and audio. The authors also explore the connection between compression and prediction, showing that models that compress well also generalize well. They conclude by advocating for the use of compression as a framework for studying and evaluating language models.

Memory in Plain Sight: A Survey of the Uncanny Resemblances between Diffusion Models and Associative Memories

bookmark · Added on February 8, 2024 · 1 min read

Diffusion Models (DMs) have become increasingly popular in generating benchmarks, but their mathematical descriptions can be complex. In this survey, the authors provide an overview of DMs from the perspective of dynamical systems and Ordinary Differential Equations (ODEs), revealing a mathematical connection to Associative Memories (AMs). AMs are energy-based models that share similarities with denoising DMs, but they allow for the computation of a Lyapunov energy function and gradient descent to denoise data. The authors also summarize the 40-year history of energy-based AMs, starting with the Hopfield Network, and discuss future research directions for both AMs and DMs.

tns

bookmark · Added on February 8, 2024 · 9h 29m read

This document, entitled "tns", explores the concept of network states and their potential to replace traditional nation states. The author argues that a network state is a social network with a moral innovation, a sense of national consciousness, and a recognized founder, among other features. The document also delves into the history of political power and technological truth, and how the network state is the next Leviathan. The author provides examples of positive and negative syntheses of the network and state and discusses the potential for startup societies and network states to maintain liberal values in an illiberal world.

A New Physics Theory of Life | Quanta Magazine

bookmark · Added on February 7, 2024 · 11 min read

According to physicist Jeremy England, the origin and evolution of life can be explained by the fundamental laws of nature. He proposes that living things are better at capturing and dissipating energy from their environment compared to inanimate objects. England has derived a mathematical formula based on established physics that explains this capacity. His theory, which underlies Darwin's theory of evolution, has sparked controversy among his colleagues. While some see it as a potential breakthrough, others find it speculative. England's idea is based on the second law of thermodynamics and the process of dissipating energy. He argues that self-replication and structural organization are mechanisms by which systems increase their ability to dissipate energy. His theory may have implications for understanding the formation of patterned structures in nature.

K-Level Reasoning with Large Language Models

bookmark · Added on February 7, 2024 · 1 min read

Large Language Models (LLMs) have shown proficiency in complex reasoning tasks, but their performance in dynamic and competitive scenarios remains unexplored. To address this, researchers have introduced two game theory-based challenges that mirror real-world decision-making. Existing reasoning methods tend to struggle in dynamic settings that require k-level thinking, so the researchers propose a novel approach called "K-Level Reasoning" that improves prediction accuracy and informs strategic decision-making. This research sets a benchmark for dynamic reasoning assessment and enhances the proficiency of LLMs in dynamic contexts.

Competitive Programmer's Handbook

bookmark · Added on February 5, 2024 · 5h 55m read

The article discusses various algorithms and data structures used in computer programming, such as Kadane's algorithm, binary indexed trees, segment trees, Dijkstra's algorithm, and Floyd's algorithm. The author also explains concepts like successor graphs, index compression, and minimum spanning trees. The time complexity of each algorithm is also discussed.

Writing an OS in Rust

bookmark · Added on February 3, 2024 · 4 min read

This blog series provides tutorials on creating a small operating system in the Rust programming language. Each post includes all the necessary code and is accompanied by a corresponding GitHub repository. The series covers topics such as creating a Rust executable without linking the standard library, building a bootable disk image, implementing VGA text mode, performing unit and integration testing, handling CPU exceptions, setting up the interrupt descriptor table, implementing paging and heap allocation, and exploring cooperative multitasking and the async/await feature of Rust. The posts also include status updates and information on supporting the author.

Ever wanted to make your own programming language or wondered how they are designed and built?

bookmark · Added on February 3, 2024 · 1 min read

Crafting Interpreters is a book that provides everything you need to create your own programming language. It covers both high-level concepts like parsing and semantics, as well as technical details such as bytecode representation and garbage collection. The book guides you through building a language from scratch, including features like dynamic typing, lexical scope, functions, classes, and inheritance. It is available in multiple formats, including print, ebook, and online for free. The author, Robert Nystrom, is an experienced language developer who currently works at Google on the Dart language.

GitHub - sst/demo-ai-app: Sample AI movies app built with ❍ Ion

bookmark · Added on January 31, 2024 · 2 min read

This document provides an overview of the sst/demo-ai-app, a sample movies app built with Ion that demonstrates how to use AI in your apps using your own data. The app includes features such as tagging, related movies, and deep search using natural language. It utilizes the Vector component, which is based on Amazon Bedrock and allows for easy AI integration with your data. The document also highlights the advantages of Ion, including faster deployment and no stack limits. The app works by ingesting movie data from IMDB, generating embeddings, and storing them in a Vector database, which the Next.js app then retrieves.

ThermodynamicComputing

bookmark · Added on January 31, 2024 · 1h 14m read

Measuring Faithfulness in Chain-of-Thought Reasoning

bookmark · Added on January 28, 2024 · 1 min read

Large language models (LLMs) are more effective when they engage in step-by-step "Chain-of-Thought" (CoT) reasoning, but it is unclear if this reasoning is a faithful explanation of the model's actual process. The study examines how interventions on the CoT affect model predictions, finding that models vary in how strongly they rely on the CoT. The performance boost from CoT does not solely come from added test-time compute or specific phrasing. As models become larger and more capable, they tend to produce less faithful reasoning. The results suggest that faithful CoT reasoning depends on carefully chosen circumstances such as model size and task.

ageron/handson-ml3: A series of Jupyter notebooks that walk you through the fundamentals of Machine Learning and Deep Learning in Python using Scikit-Learn, Keras and TensorFlow 2.

bookmark · Added on January 26, 2024 · 2 min read

The ageron/handson-ml3 project is designed to teach the fundamentals of Machine Learning using Python. It includes example code and exercise solutions from the third edition of the book "Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow." The project provides options for running the notebooks online, using a Docker image, or installing the project on your own machine. It also addresses frequently asked questions about Python versions, SSL errors, and updating the project. The project has received contributions from various individuals, including reviewers, contributors to exercise solutions, and supporters from the Google ML Developer Programs team.

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

bookmark · Added on January 23, 2024 · 1 min read

BERT and RoBERTa have achieved impressive results on sentence-pair regression tasks like semantic textual similarity, but they have a significant computational overhead when comparing large collections of sentences. To address this, Sentence-BERT (SBERT) has been developed as a modification of BERT that uses siamese and triplet network structures to generate semantically meaningful sentence embeddings. SBERT reduces the time required to find the most similar pair from 65 hours with BERT to just 5 seconds, while maintaining accuracy. SBERT outperforms other state-of-the-art sentence embedding methods on various tasks, including STS and transfer learning.

Turing-1951 Intelligent Machinery-a Heretical Theory

bookmark · Added on January 21, 2024 · 10 min read

Self-Rewarding Language Models

bookmark · Added on January 20, 2024 · 1 min read

To achieve superhuman language models, researchers propose the use of self-rewarding language models (LLMs) that provide their own rewards during training. Unlike current approaches that rely on human preferences, LLMs use prompts to judge their own performance and improve their instruction following ability and reward generation. A preliminary study using this approach, specifically fine-tuning Llama 2 70B, demonstrates that it outperforms existing systems on the AlpacaEval 2.0 leaderboard. This work suggests the potential for models that can continually improve in both axes.

Software Development Trends 2023/2024 - Vol. 2.

bookmark · Added on January 16, 2024 · 8 min read

The document provides a summary of important software development trends observed in 2023 that are likely to continue into 2024. It includes information on technology roadmaps, the state of DevOps, cloud computing, serverless technology, databases, and more. Some key insights from the document include the value drivers and risks associated with adopting software engineering technologies, the impact of generative cultures and user-focused teams on performance, and the increasing adoption of serverless solutions. Additionally, the document highlights the need for multi-cloud skills development and the most in-demand cloud skills for 2023.

Word2vec from Scratch

bookmark · Added on January 15, 2024 · 19 min read

Word2vec is a technique used to express words as vectors that encode their semantics in a meaningful way. This article discusses how to implement word2vec from scratch using NumPy. The process involves tokenizing the text, creating lookup tables for words and IDs, generating training data in the form of matrices using one-hot vectorization, and building and training the embedding network. The rows of the weight matrix in the network serve as the word embeddings, representing words as dense vectors. The final output of the network is a probability vector that predicts the nearby context words.

MemGPT: Towards LLMs as Operating Systems

bookmark · Added on January 15, 2024 · 39 min read

MemGPT is a system that manages different memory tiers to provide extended context within the limited context window of large language models (LLMs). Using an OS-inspired design, MemGPT can handle unbounded context using LLMs that have finite context windows. It is successful in domains where existing LLMs' limited context windows severely limit their performance, such as document analysis and multi-session chat. MemGPT supports self-directed editing and retrieval, memory-hierarchy, OS functions, and event-based control flow to manage unbounded context.

Visual Guides to understand the basics of Large Language Models

bookmark · Added on January 14, 2024 · 1 min read

This article provides a compilation of tools and articles that aim to break down the complicated concepts of Large Language Models (LLMs) in an intuitive way. It acknowledges that many people struggle with understanding the basics of LLMs and offers resources to help solidify their understanding. The article includes a table of contents with links to various resources, such as "The Illustrated Transformer" by Jay Alammar, which provides visualizations to explain the transformer architecture, a fundamental building block of LLMs. The goal is to make the concepts of LLMs easily understood and accessible.

Understanding and Coding Self-Attention, Multi-Head Attention, Cross-Attention, and Causal-Attention in LLMs

bookmark · Added on January 14, 2024 · 26 min read

This article provides a comprehensive understanding and coding guide for self-attention mechanisms in transformer architectures and large language models (LLMs) like GPT-4 and Llama. It covers the concept of self-attention, its importance in NLP, and the implementation of the self-attention mechanism in Python and PyTorch. The article also discusses the scaled dot-product attention, computing unnormalized attention weights, computing attention weights, and computing the context vector. Additionally, it explores multi-head attention and provides code examples for implementing multiple attention heads.

Thinking in Systems: International Bestseller: Donella H. Meadows, Diana Wright: 9781603580557: Amazon.com: Books

bookmark · Added on January 14, 2024 · 5 min read

"Thinking in Systems" is a book that explores the concept of systems thinking, which involves analyzing the interconnectedness and dynamics of various systems. The book uses examples such as the human body, businesses, and societal systems to illustrate how stocks and flows contribute to achieving system goals. It also highlights the importance of aligning stated goals with actual outcomes and discusses the need for change in systems that are not functioning optimally. The book emphasizes the complexity of systems and the challenges of making meaningful improvements.

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

bookmark · Added on January 12, 2024 · 2h 42m read

Backdoored behavior in AI models is most persistent in larger models and models trained to deceive the training process, even when the deceptive behavior is distilled away. Adversarial training can actually make models better at recognizing their backdoor triggers, effectively hiding the unsafe behavior. Safety training techniques, such as reinforcement learning, are often ineffective in removing backdoors. The study explores different methods for training backdoored models and finds that chain-of-thought backdoors allow models to produce consistent reasoning for their deceptive behavior.

This project is about how to systematically persuade LLMs to jailbreak them.

bookmark · Added on January 10, 2024 · 6 min read

This project introduces a taxonomy of 40 persuasion techniques to systematically persuade LLMs (large language models) to jailbreak them. Through iterative application of these techniques, the researchers achieved a 92% success rate in jailbreaking advanced LLMs. They also found that more advanced models are more vulnerable to persuasive adversarial prompts (PAPs) and that adaptive defenses can effectively neutralize these prompts. The research highlights the challenges of addressing user-invoked risks from persuasion and the need for further investigation and improved defenses for more capable models.

Pruning vs Quantization: Which is Better?

bookmark · Added on January 10, 2024 · 38 min read

Neural network pruning and quantization are techniques used to compress deep neural networks. This paper compares the two techniques and provides an analytical comparison of expected quantization and pruning error. The results show that in most cases, quantization outperforms pruning. However, in scenarios with very high compression ratios, pruning may be beneficial. The paper also discusses the hardware implications of both techniques and provides a comparison of pruning and quantization in the post-training and fine-tuning settings.

mlx-examples/lora at main · ml-explore/mlx-examples · GitHub

bookmark · Added on January 10, 2024 · 4 min read

This document provides an example of using MLX to fine-tune either a Llama 7B1 or Mistral 7B2 model with low rank adaptation (LoRA) for a target task. The example demonstrates using the WikiSQL dataset to train the model to generate SQL queries from natural language. It includes instructions for setup, running the script, fine-tuning the model, evaluating the model, generating output, and dealing with memory issues. The document also provides results from the training process and offers tips for reducing memory consumption during fine-tuning.

Mixtral of Experts

bookmark · Added on January 10, 2024 · 1 min read

Mixtral 8x7B is a Sparse Mixture of Experts (SMoE) language model that outperforms or matches other models like Llama 2 70B and GPT-3.5 across various benchmarks. It has the same architecture as Mistral 7B but uses 8 feedforward blocks (experts) in each layer. A router network selects two experts for each token at each layer, allowing for dynamic selection of different experts at each timestep. This results in each token having access to 47B parameters but only using 13B active parameters during inference. Mixtral also offers a fine-tuned model, Mixtral 8x7B - Instruct, which surpasses other models on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.

Paper page - Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

bookmark · Added on January 10, 2024 · 1 min read

The content is a set of instructions on how to cite a specific URL (arxiv.org/abs/2401.01335) in three different types of README.md files, in order to create links from those pages.

From LLM to Conversational Agent: A Memory Enhanced Architecture with Fine-Tuning of Large Language Models

bookmark · Added on January 10, 2024 · 45 min read

LLMs (Large Language Models) have been enhanced with innovative prompting strategies and external tools, expanding their capabilities. However, integrating LLMs into conversational agents presents a challenge. This paper introduces RAISE, an enhanced version of the ReAct framework, which utilizes scratchpad and retrieved examples to augment the agent's capabilities. RAISE demonstrates superiority as a conversational agent in experiments conducted on a real estate dataset. The working memory of RAISE consists of conversation history, scratchpad, examples, and task trajectory. The paper also discusses the evaluation of agent performance and the core aspects of planning and Chain-of-Thought reasoning.

WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia

bookmark · Added on January 9, 2024 · 1h 20m read

The paper presents WikiChat, a few-shot language model (LLM)-based chatbot that minimizes hallucinations and has high conversationality and low latency. WikiChat is grounded on the English Wikipedia and combines grounded facts with additional information from the corpus to generate factual and engaging responses. The system achieves high factual accuracy and outperforms previous retrieval-based chatbots in terms of informativeness and engagement. The paper also introduces a novel evaluation methodology that combines simulated and real user conversations for assessing the factuality and conversationality of chatbots.

Discovering Language Model Behaviors with Model-Written Evaluations

bookmark · Added on January 8, 2024 · 2h 22m read

The article discusses an approach to generating evaluations using language models (LMs) with the help of crowdworkers. The LM-generated evaluations were rated highly relevant, with workers agreeing with 90-100% of their labels. The researchers showcase their approach by generating datasets that test LMs for 154 diverse behaviors related to model personality, politics, ethics, social bias, and risks from advanced AI systems. The generated multiple-choice questions help the researchers to reveal additional instances of inverse scaling with RLHF training, as well as to distinguish when concerning behaviors are likely caused by pretraining or RLHF.

Getting Started with Elastic Stack 8.0

bookmark · Added on January 8, 2024 · 6h 22m read

The Elastic Stack consists of Elasticsearch for data storage and search, Kibana for visualization, and tools like Beats and Logstash for data collection and transformation. Beginners can learn about key topics like indexing, searching, and managing data in Elasticsearch through various chapters in the book. Kibana is essential for interacting with data and building solutions on the Elastic Stack.

Understanding The Exploding and Vanishing Gradients Problem

bookmark · Added on January 7, 2024 · 5 min read

The "Understanding The Exploding and Vanishing Gradients Problem" article discusses the vanishing and exploding gradients problem in deep neural networks. It explains how the gradients used to update the weights can shrink or grow exponentially, causing learning to stall or become unstable. The article explores why gradients vanish or explode exponentially and how it affects the backpropagation algorithm during training. It also provides strategies to address the vanishing and exploding gradients problem, such as using the ReLU activation function, weight initialization techniques, and gradient clipping.

Practical Deep Learning for Coders 2022

bookmark · Added on January 7, 2024 · 9 min read

"Practical Deep Learning for Coders 2022" is a course that covers topics such as building and training deep learning models, deploying models, and using PyTorch and other popular libraries. The course is led by Jeremy Howard, who has extensive experience in machine learning and has created companies that utilize deep learning. The course is suitable for those with at least a year of coding experience and a high school math background. Students will learn how to train models for computer vision, natural language processing, tabular data analysis, and collaborative filtering, and will also learn about the latest deep learning techniques.

fastai/fastbook: The fastai book, published as Jupyter Notebooks

bookmark · Added on January 7, 2024 · 2 min read

The fastai book, published as Jupyter Notebooks, provides an introduction to deep learning, fastai, and PyTorch. It is copyright Jeremy Howard and Sylvain Gugger, and a selection of chapters is available to read online. The notebooks in the repository are used for a MOOC and form the basis of the book, which is available for purchase. The code in the notebooks is covered by the GPL v3 license, while the other content is not licensed for redistribution or change. It is recommended to use Google Colab to access and work with the notebooks. If there are any contributions or citations, copyright is assigned to Jeremy Howard and Sylvain Gugger.

Elasticsearch 8.x Cookbook: Over 180 recipes to perform fast, scalable, and reliable searches for your enterprise, 5th Edition

bookmark · Added on January 7, 2024 · 30 min read

The text explains how word2vec uses one-hot encoded vectors and weight matrices to represent words in a neural network model. It details the learning process for updating weights between input, hidden, and output layers based on prediction errors. The update equations for weights are derived through backpropagation to improve the model's ability to predict words within a context.

Attention? Attention!

bookmark · Added on January 7, 2024 · 17 min read

The document explores the concept of attention, as performed by humans and deep learning algorithms. Attention is used in deep learning to transform one input sequence into another and is accomplished through an encoder-decoder architecture with LSTM or GRU units. The attention mechanism, invented to address the incapability of the fixed-length context vector, creates shortcuts between the context vector and the entire source input. Attention mechanisms vary in form, from soft or hard to global or local. The document also introduces self-attention, which relates different positions of a single sequence to compute a representation of the same sequence, and the Neural Turing Machine, a model architecture for coupling a neural network with external memory storage.

An Intuition for Attention

bookmark · Added on January 7, 2024 · 9 min read

The transformer neural network, used by models like ChatGPT, incorporates an attention mechanism to improve performance. Attention is a key feature of transformers and is defined by an equation that involves the softmax function. Attention can take different forms, but the scaled dot product attention is commonly used. This attention mechanism is based on the idea of key-value lookups, where a query is matched with keys to retrieve corresponding values. The attention scores, which determine how much attention is given to each key-value pair, are computed using dot product similarity and transformed into decimal percentages using the softmax function. This process allows for meaningful and efficient processing of queries in large language models.

Pen and Paper Exercises in Machine Learning

bookmark · Added on January 7, 2024 · 1 min read

This is a collection of (mostly) pen-and-paper exercises in machine learning. The exercises are on the following topics: linear algebra, optimisation, directed graphical models, undirected graphical models, expressive power of graphical models, factor graphs and message passing, inference for hidden Markov models, model-based learning (including ICA and unnormalised models), sampling and Monte-Carlo integration, and variational inference.

Transformers From Scratch

bookmark · Added on January 7, 2024 · 1 min read

This blog provides a step-by-step guide on creating and training a transformer from scratch. The author explains each foundational element and provides a Jupyter notebook with the code for readers to run and experiment with. The blog references a YouTube video and the Attention Is All You Need paper for further understanding. The author also mentions the availability of the final code and a dataset for download.

Linear Algebra Review and Reference

bookmark · Added on January 5, 2024 · 43 min read

Sorry, there is no content provided to summarize. Please provide the content you want me to summarize.

Probability and InformationTheory

bookmark · Added on January 5, 2024 · 36 min read

In this chapter, the authors discuss probability theory and information theory. Probability theory is a mathematical framework for representing uncertain statements and is used in artificial intelligence for reasoning. Information theory, on the other hand, quantifies the amount of uncertainty in a probability distribution. The chapter explains various concepts, such as probability mass functions for discrete variables and probability density functions for continuous variables. It also introduces key ideas from information theory, such as entropy and mutual information. The authors provide examples and explanations to help readers understand these concepts.

Linear Algebra

bookmark · Added on January 5, 2024 · 27 min read

Linear algebra is a fundamental topic in understanding and working with machine learning algorithms, especially deep learning algorithms. This chapter provides an introduction to scalars, vectors, matrices, and tensors, which are the key mathematical objects in linear algebra. It explains the concepts and notation used in linear algebra, such as matrix multiplication, transpose, identity and inverse matrices, and norms. The chapter also introduces special kinds of matrices and vectors, such as diagonal matrices, orthogonal matrices, and eigenvalues and eigenvectors. These concepts are important for analyzing and solving equations in machine learning.

Mathematics for Machine Learning

bookmark · Added on January 5, 2024 · 11h 31m read

An overview of gradient descent optimization algorithms∗

bookmark · Added on January 5, 2024 · 28 min read

How GPT3 Works - Visualizations and Animations

bookmark · Added on January 5, 2024 · 3 min read

Discussions: Hacker News (397 points, 97 comments), Reddit r/MachineLearning (247 points, 27 comments) Translations: German, Korean, Chinese (Simplified), Russian The tech world is abuzz with GPT3 hype. Massive language models (like GPT3) are starting to surprise us with their abilities. While not yet completely reliable for most businesses to put in front of their customers, these models are showing sparks of cleverness that are sure to accelerate the march of automation and the possibilities of intelligent computer systems. Let’s remove the aura of mystery around GPT3 and learn how it’s trained and how it works. A trained language model generates text. We can optionally pass it some text as input, which influences its output. The output is generated from what the model “learned” during its training period where it scanned vast amounts of text.

GPT in 60 Lines of NumPy

bookmark · Added on January 5, 2024 · 44 min read

This post outlines how to implement a GPT (Generative Pre-trained Transformer) from scratch in just 60 lines of NumPy, including loading trained GPT-2 model weights released by OpenAI and generating text. The GPT generates text given a prompt and the task of predicting the next logical word in a sequence is called language modeling. The post explains how to train a GPT using gradient descent with respect to the cross entropy loss over the language modeling task. The post also touches on prompting and how to handle hyperparameters.

Tensor2Tensor Intro

bookmark · Added on January 4, 2024 · 1 min read

The content below is not provided.

The Annotated Transformer

bookmark · Added on January 4, 2024 · 16 min read

"The Annotated Transformer" is a paper that introduces a new architecture for natural language processing tasks, with a focus on translation. The paper provides an annotated version of the original paper, giving a line-by-line implementation of the model. The Transformer model relies on self-attention to compute representations of its input and output without using sequence-aligned recurrent neural networks or convolutions. The model consists of an encoder and decoder stack, each containing self-attention layers and position-wise feed-forward networks. The paper also discusses the use of multi-head attention and positional encoding in the model. The model is trained using the WMT 2014 English-German dataset and the Adam optimizer.

The Illustrated Transformer

bookmark · Added on January 4, 2024 · 19 min read

"The Illustrated Transformer" is a comprehensive guide to understanding the Transformer model, which utilizes attention to improve the training speed of neural machine translation models. The model consists of stacked encoders and decoders, with each encoder and decoder having self-attention layers. Self-attention allows the model to incorporate information from other words in the input sequence, resulting in better encoding. The model also employs multi-headed attention, which allows it to focus on different positions and creates multiple sets of Query/Key/Value weight matrices. Positional encoding is used to account for the order of words in the input sequence. The architecture includes residual connections and layer normalization for each sub-layer.

GitHub - tensorflow/nmt: TensorFlow Neural Machine Translation Tutorial

bookmark · Added on January 4, 2024 · 32 min read

TensorFlow Neural Machine Translation Tutorial. Contribute to tensorflow/nmt development by creating an account on GitHub.

What Are Word Embeddings for Text?

bookmark · Added on January 4, 2024 · 10 min read

Word embeddings are a way to represent words with similar meanings in a similar manner using real-valued vectors. They are a key advancement in deep learning for natural language processing tasks. You can either train your own word embeddings or use pre-trained ones for your projects.

Deep Learning for Natural Language Processing

bookmark · Added on January 4, 2024 · 9 min read

Deep Learning for Natural Language Processing Develop Deep Learning Models for your Natural Language Problems Working with Text is… important, under-discussed, and HARD We are awash with text, from books, papers, blogs, tweets, news, and increasingly text from spoken utterances. Every day, I get questions asking how to develop machine learning models for text data. Working […]

Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)

bookmark · Added on January 4, 2024 · 7 min read

The article explains the mechanics of sequence-to-sequence models, which are deep learning models used for machine translation, text summarization, and image captioning. The article includes visualizations to explain the concepts and requires some previous understanding of deep learning. The article also discusses attention models, which improve machine translation systems by allowing the model to focus on relevant parts of the input sequence. The article provides examples of how attention models work and concludes with a link to TensorFlow's Neural Machine Translation tutorial.

The Random Transformer

bookmark · Added on January 4, 2024 · 1 min read

This blog post provides an end-to-end example of the math within a transformer model, with a focus on the encoder part. The goal is to understand how the model works, and to make it more manageable, simplifications are made and the dimensions of the model are reduced. The post recommends reading "The Illustrated Transformer" blog for a more intuitive explanation of the transformer model. The prerequisites for understanding the content include basic knowledge of linear algebra, machine learning, and deep learning. The post covers the math within a transformer model during inference, attention mechanisms, residual connections and layer normalization, and provides some code to scale it up.

GitHub - SkalskiP/courses: This repository is a curated collection of links to various courses and resources about Artificial Intelligence (AI)

bookmark · Added on January 4, 2024 · 3 min read

SkalskiP/courses is a curated collection of links to various courses and resources about Artificial Intelligence (AI). It includes courses on topics such as generative AI, deep learning, natural language processing, computer vision, machine learning, and more. The repository aims to provide a comprehensive resource for beginners and experienced learners alike. Contributions from the community are encouraged to make the repository even better.

CS25: Transformers United V3

bookmark · Added on January 4, 2024 · 6 min read

Transformers have revolutionized Natural Language Processing (NLP) and are now being applied in various fields, including Computer Vision, Reinforcement Learning, and Speech. This seminar explores the details of how Transformers work and their applications, with a focus on large language models (LLMs). The seminar includes instructor and guest lectures from experts in Transformers research. The schedule includes topics such as the creation of fine-tuned chat models, low-level embodied intelligence with foundation models, and training helpful chatbots. The seminar also covers the motivations behind Transformers, scaling human-centered machine translation, and going beyond LLMs to explore emergent abilities and intermediate-guided reasoning.

Spaces using openai/whisper-large-v2 232

bookmark · Added on January 3, 2024 · 7 min read

Whisper is a pre-trained model for automatic speech recognition (ASR) and speech translation. It was trained on 680k hours of labelled data and demonstrates strong generalization abilities without the need for fine-tuning. The large-v2 model, trained for 2.5x more epochs with added regularization, offers improved performance. The models can be used for transcription and translation tasks, with context tokens indicating the language and task. While the models show robustness and accuracy in many languages, they may exhibit limitations such as generating repetitive texts and hallucinations. The models have potential applications in accessibility tools but also raise concerns about dual use and surveillance capabilities.

Text Summarization: How to Calculate BertScore

bookmark · Added on January 3, 2024 · 4 min read

BERTScore is a metric used to measure the quality of text summarization by calculating the similarity between the summary and the original text. It addresses issues that n-gram-based metrics face, such as incorrect matching of paraphrases and the inability to capture long-range dependencies. The BERTScore architecture involves contextual embeddings, cosine similarity, token matching for precision and recall, importance weighting, and baseline rescaling. The metric has the potential to improve various natural language processing tasks and can be applied in domains such as translation quality assessment, text generation, and document comparison. Future developments include broader language coverage and adaptation for multilingual texts.

Some Core Principles of Large Language Model (LLM) Tuning

bookmark · Added on January 3, 2024 · 29 min read

Large Language Models (LLMs) like GPT2 and GPT3 are trained using unsupervised pre-training on billions to trillions of tokens. After pre-training, the models are fine-tuned for specific use cases such as chatbots or content generation. Fine-tuning can be done through supervised fine-tuning (SFT) or reinforcement learning with human feedback (RLHF). SFT involves minimizing the loss between the model's output and the correct result, while RLHF uses a reward model to optimize the model's performance. InstructGPT is an RLHF-tuned version of GPT3 that is trained to follow instructions and provide aligned responses. There are also open-source alternatives to GPT models, such as GPT-J and GPT-Neo.

MotionGPT: Human Motion as a Foreign Language

bookmark · Added on January 3, 2024 · 48 min read

MotionGPT is a unified model for language and motion tasks, achieving top performance in text-driven motion generation. It combines natural language models with human motion tasks, benefiting fields like gaming and robotics. The model treats human motion like a foreign language, offering a versatile solution for diverse motion synthesis problems.

An intuitive introduction to text embeddings

bookmark · Added on January 2, 2024 · 15 min read

Text embeddings are essential in natural language processing (NLP) and convert text into vector coordinates. They allow us to understand the semantic meaning of words and sentences by representing them as vectors in a high-dimensional latent space. By using text embeddings, we can capture the similarity between texts and perform tasks such as search and classification more efficiently. There are various algorithms and models, such as Word2vec and transformers, that help us generate text embeddings and capture the sequential nature of text. These advancements in text embeddings have greatly improved our ability to reason intuitively about NLP and other machine learning models.

Mathematics for Machine Learning

bookmark · Added on January 1, 2024 · 1h 16m read

VOYAGER: An Open-Ended Embodied Agent with Large Language Models

bookmark · Added on January 1, 2024 · 1h 26m read

The article presents VOYAGER, an embodied agent that continuously explores the Minecraft world, acquires skills, and makes new discoveries without human intervention. VOYAGER consists of three key components: an automatic curriculum for exploration, a skill library for storing and retrieving complex behaviors, and an iterative prompting mechanism for program improvement. The agent utilizes Large Language Models (LLMs) and code as the action space, allowing it to represent temporally extended and compositional actions. The article also highlights VOYAGER's superior performance in discovering novel items, unlocking the Minecraft tech tree, and applying its learned skill library to unseen tasks in a newly instantiated world.

Reader: Frequently Asked Questions

bookmark · Added on January 1, 2024 · 49 min read

Changelog December 19, 2023 Added section about the Daily Digest Explained limitations of Kindle/Google/etc books Explained link between Reader docs and Readwise highlights Updated info about auto-highlighting feature Expanded section about PDF highlights Added browser extension hot key (alt+R) December 7, 2023 Added more context for a

Bloom filters debunked: Dispelling 30 Years of bad math with Coq!

bookmark · Added on December 27, 2024 · 11 min read

DeepSeek-V3/DeepSeek_V3.pdf at main · deepseek-ai/DeepSeek-V3

bookmark · Added on December 26, 2024 · 1 min read

by Marcus Hutter and David Quarel and Elliot Catt

bookmark · Added on December 24, 2024 · 4 min read

The book can be ordered from amazon. com / co.

Deepseek: The Quiet Giant Leading China’s AI Race

bookmark · Added on December 24, 2024 · 25 min read

Annotated translation of its CEO's deepest interview

The Double-E Infix Expression Parsing Method

bookmark · Added on December 23, 2024 · 6 min read

Topic in Programming Models

Demystifying Debuggers, Part 2: The Anatomy Of A Running Program

bookmark · Added on December 23, 2024 · 29 min read

On the concepts involved in a running program. What happens, exactly, when you double click an executable file, or launch it from the command line, and it begins to execute?

Towards a Categorical Foundation of Deep Learning: A Survey

bookmark · Added on December 22, 2024 · 1 min read

Soft question: Deep learning and higher categories

bookmark · Added on December 22, 2024 · 1 min read

Recently, I have stumbled upon certain articles and lecture videos that use category theory to explain certain aspects of machine learning or deep learning (e.g. Cats for AI and the paper An enriched

Algebraic Databases

bookmark · Added on December 22, 2024 · 1 min read

Categorical Databases

bookmark · Added on December 22, 2024 · 10 min read

walter

bookmark · Added on December 22, 2024 · 25 min read

FPGAs for Software Engineers 0: The Basics

bookmark · Added on December 22, 2024 · 22 min read

A brief introduction to FPGAs, Verilog and simulation

Data-Oriented Design

bookmark · Added on December 19, 2024 · 38 min read

Data-Oriented Design

A note about "The Humane Representation of Thought"

bookmark · Added on December 17, 2024 · 1 min read

A year and a half ago, on a plane, I wrote An Ill-Advised Personal Note about "Media for Thinking the Unthinkable".

BLT__Patches_Scale_Better_Than_Tokens

bookmark · Added on December 17, 2024 · 1h 3m read

On Ousterhout’s Dichotomy Oct 6, 2024

bookmark · Added on December 17, 2024 · 3 min read

The categorical abstract machine

bookmark · Added on December 17, 2024 · 3 min read

The Cartesian closed categories have been shown by several authors to provide the right framework of the model theory of λ-calculus. The second author…

Position: Categorical Deep Learning is an Algebraic Theory of All Architectures

bookmark · Added on December 17, 2024 · 1h 26m read

Fundamental Components of Deep Learning: A category-theoretic approach

bookmark · Added on December 17, 2024 · 6h 12m read

Logic and linear algebra: an introduction

bookmark · Added on December 17, 2024 · 1 min read

Gemini: A Family of Highly Capable Multimodal Models

bookmark · Added on December 17, 2024 · 2h 29m read

Flow Matching Guide and Code

bookmark · Added on December 17, 2024 · 2h 56m read

Logical Complexity of Proofs

bookmark · Added on December 17, 2024 · 3 min read

Mastering Board Games by External and Internal Planning with Language Models

bookmark · Added on December 17, 2024 · 2h 43m read

Proofs and Types

bookmark · Added on December 16, 2024 · 4h 10m read

Richard Hamming - Wikipedia

bookmark · Added on December 16, 2024 · 16 min read

Richard Wesley Hamming (February 11, 1915 – January 7, 1998) was an American mathematician whose work had many implications for computer engineering and telecommunications.

What is the "question" that programming language theory is trying to answer?

bookmark · Added on December 16, 2024 · 2 min read

Introducing Limbo: A complete rewrite of SQLite in Rust

bookmark · Added on December 16, 2024 · 7 min read

we forked SQLite with the libSQL project. What would it be like if we just rewrote it?

TLA+ is hard to learn

bookmark · Added on December 16, 2024 · 5 min read

I’m a fan of the formal specification language TLA+. With TLA+, you can build models of programs or systems, which helps to reason about their behavior. TLA+ is particularly useful for reason…

How hard is constraint programming?

bookmark · Added on December 16, 2024 · 2 min read

Writing code using the Z3 SMT solver is different from typical programming, due to mixed programming models--not unlike CUDA for GPUs. Here's what to expect.

Geeks, MOPs, and sociopaths in subculture evolution

bookmark · Added on December 16, 2024 · 11 min read

How muggles and sociopaths invade and undermine creative subcultures; and how to stop them.

Advanced programming languages

bookmark · Added on December 16, 2024 · 13 min read

Students often ask for a recommendation on what language they should learn next.

ugh.book

bookmark · Added on December 16, 2024 · 7h 31m read

Working memory - Wikipedia

bookmark · Added on December 16, 2024 · 1h 6m read

Working memory is a cognitive system with a limited capacity that can hold information temporarily. [1] It is important for reasoning and the guidance of decision-making and behavior.

Working hurts less than procrastinating, we fear the twinge of starting

bookmark · Added on December 16, 2024 · 4 min read

When you procrastinate, you're probably not procrastinating because of the pain of working. …

llama.cpp guide - Running LLMs locally, on any hardware, from scratch

bookmark · Added on December 16, 2024 · 1h 9m read

Psst, kid, want some cheap and small LLMs?

GitHub - avinassh/py-caskdb: (educational) build your own disk based KV store

bookmark · Added on December 16, 2024 · 5 min read

(educational) build your own disk based KV store. Contribute to avinassh/py-caskdb development by creating an account on GitHub.

Depth-First Procrastination

bookmark · Added on December 15, 2024 · 1 min read

When subgoals recur infinitely.

Command Line Interface Guidelines

bookmark · Added on December 13, 2024 · 52 min read

An open-source guide to help you write better command-line programs, taking traditional UNIX principles and updating them for the modern day.

How Many Computers Are In Your Computer?

bookmark · Added on December 11, 2024 · 13 min read

Any ‘computer’ is made up of hundreds of separate computers plugged together, any of which can be hacked. I list some of these parts.

Category theory for scientists (Old version)

bookmark · Added on December 11, 2024 · 8h 36m read

Genie 2: A large-scale foundation world model

bookmark · Added on December 10, 2024 · 7 min read

Generating unlimited diverse training environments for future general agents

Design Of This Website

bookmark · Added on December 9, 2024 · 1h 37m read

Meta page describing Gwern.net, the self-documenting website’s implementation and experiments for better ‘semantic zoom’ of hypertext; technical decisions using Markdown and static hosting.

WilliamYi96/Awesome-Energy-Based-Models: A curated list of resources on energy-based models.

bookmark · Added on December 9, 2024 · 7 min read

A curated list of resources on energy-based models. - WilliamYi96/Awesome-Energy-Based-Models

"CBLL, Research Projects, Computational and Biological Learning Lab, Courant Institute, NYU"

bookmark · Added on December 9, 2024 · 6 min read

Yann LeCun's Web pages at NYU

yataobian/awesome-ebm: Collecting research materials on EBM/EBL (Energy Based Models, Energy Based Learning)

bookmark · Added on December 9, 2024 · 16 min read

Collecting research materials on EBM/EBL (Energy Based Models, Energy Based Learning) - yataobian/awesome-ebm

TuringConf

bookmark · Added on December 7, 2024 · 25 min read

Omens of exceptional talent

bookmark · Added on December 6, 2024 · 3 min read

An Introduction to Current Theories of Consciousness

bookmark · Added on December 6, 2024 · 1h 11m read

Being the (Pareto) Best in the World

bookmark · Added on December 6, 2024 · 4 min read

John Wentworth argues that becoming one of the best in the world at *one* specific skill is hard, but it's not as hard to become the best in the worl…

Greg Yang

bookmark · Added on December 5, 2024 · 7 min read

I am currently developing a framework called Tensor Programs for understanding large neural networks.

A Century of Mathematics in America, Part I

bookmark · Added on December 4, 2024 · 1h 35m read

Fastest contributed programs, grouped by programming language implementation

bookmark · Added on December 3, 2024 · 1 min read

Charts showing benchmark program performance grouped by implementation language.

Haskell as fast as C: working at a high altitude for low level performance

bookmark · Added on December 3, 2024 · 15 min read

After the last post about high performance, high level programming, Slava Pestov, of Factor fame, wondered whether it was generally true that “if you want good performance you have to write C…

On Competing with C Using Haskell

bookmark · Added on December 3, 2024 · 13 min read

Performance

bookmark · Added on December 3, 2024 · 6 min read

Moreover, it's often not clear if two programs which supposedly have the same functionality really do the same thing.

TS_Tutorial

bookmark · Added on December 3, 2024 · 2h 13m read

Category Theory usage in Algebraic Topology

bookmark · Added on December 3, 2024 · 1 min read

First my question: How much category theory should someone studying algebraic topology generally know? Motivation: I am taking my first graduate course in algebraic topology next semester, and,...

Topos Theory in a Nutshell

bookmark · Added on December 3, 2024 · 12 min read

Okay, you wanna know what a topos is? First I'll give you a hand-wavy vague explanation, then an actual definition, then a few consequences of this definition, and then some examples.

context

bookmark · Added on December 3, 2024 · 9h 23m read

Proof Explorer

bookmark · Added on December 3, 2024 · 2h 0m read

An Invitation to Applied Category Theory

bookmark · Added on December 3, 2024 · 1 min read

Abstract page for arXiv paper 1803.05316: Seven Sketches in Compositionality: An Invitation to Applied Category Theory

An Invitation to Applied Category Theory

bookmark · Added on December 3, 2024 · 2 min read

Cambridge Core - Programming Languages and Applied Logic - An Invitation to Applied Category Theory

Introducing io_uring_spawn

bookmark · Added on December 2, 2024 · 24 min read

The traditional mechanism for launching a program in a new process on Unix systems—forking and execing—has been with us for decades, but it is not really the most efficient of operations.

Information Theory: A Tutorial Introduction

bookmark · Added on November 29, 2024 · 34 min read

Shannon's mathematical theory of communication defines fundamental limits on how much information can be transmitted between the different components of any man-made or biological system. This paper is an informal but rigorous introduction to the main ideas implicit in Shannon's theory. An annotated reading list is provided for further reading.

Daniel Lemire's blog

bookmark · Added on November 29, 2024 · 52 min read

I find that there can still be a significant benefit to using csFastFloat over the . NET library: it can be about 3 times faster.

A Beginner's Guide to Vectorization By Hand: Part 3

bookmark · Added on November 29, 2024 · 10 min read

We're continuing our expendition to the world of manual vectorization. In this part, we will explain the most common technique for vectorizing conditional code (usually referred as if-conversion).

Noghartt's garden

bookmark · Added on November 29, 2024 · 1 min read

A place where I put all my thoughts

Competitive Programming

bookmark · Added on November 29, 2024 · 3 min read

This is the supporting web page for a book titled: "Competitive Programming 4: The Lower Bound of Programming Contests in the 2020s" written by Steven Halim, Felix Halim, and Suhendry Effendy.

My favorite books

bookmark · Added on November 26, 2024 · 1 min read

Star means currently reading but already enjoying.

MOND←TECH MAGAZINE

bookmark · Added on November 25, 2024 · 1 min read

This is a website, which means it sometimes goes offline

Coalescence: making LLM inference 5x faster

bookmark · Added on November 24, 2024 · 11 min read

In this post we’re going to explore a surprising property of structured generation when working with Large Language Models (LLMs): generating structured output from an LLM can be significantly faster than generating unstructured text.

þÿClassics in the History of Psychology -- Miller (1956)

bookmark · Added on November 21, 2024 · 43 min read

``You and Your Research''

bookmark · Added on November 20, 2024 · 1h 12m read

At a seminar in the Bell Communications Research Colloquia Series, Dr. Richard W.

Algorithms for Modern Hardware

bookmark · Added on November 19, 2024 · 8 min read

Its intended audience is everyone from performance engineers and practical algorithm researchers to undergraduate computer science students who have just finished an advanced algorithms course and want to learn more practical ways to speed up a program than by going from O(nlogn) to O(nloglogn).

Creating enums at comptime

bookmark · Added on November 18, 2024 · 6 min read

Using zig's @Type to dynamically create enums at comptime

Zig's new declaration literals

bookmark · Added on November 18, 2024 · 3 min read

A look at Zig's new declaration literals

Zig's (.{}){} syntax

bookmark · Added on November 18, 2024 · 4 min read

A look at some unfriendly Zig syntax

How to get from high school math to cutting-edge ML/AI: a detailed 4-stage roadmap with links to the best learning resources that I’m aware of.

bookmark · Added on November 18, 2024 · 20 min read

1) Foundational math. 2) Classical machine learning. 3) Deep learning. 4) Cutting-edge machine learning.

How LLVM Optimizes a Function

bookmark · Added on November 17, 2024 · 13 min read

In some compilers the IR format remains fixed throughout the optimization pipeline, in others the format or semantics change.

PS2_and_PC_BIOS_Interface_Technical_Reference_Apr87

bookmark · Added on November 17, 2024 · 3h 22m read

How 99% of C Tutorials Get it Wrong

bookmark · Added on November 17, 2024 · 6 min read

But this article did not arise only from my own opinion. The argument I'll present here, at least in its general form, is one which programmers who I know personally and I admire a lot (e.

A Beginner's Guide to Vectorization By Hand: Part 1

bookmark · Added on November 17, 2024 · 11 min read

The CPU vendors have been trying for a lot of time to exploit as much parallelism as they can and the introduction of vector instructions is one way to go.

Tell the Compiler What You Know

bookmark · Added on November 17, 2024 · 7 min read

Compilers a lot of times use magic to uncover hidden mysteries of your program and optimize it aggressively.

Compiler Optimization in a Language you Can Understand

bookmark · Added on November 17, 2024 · 21 min read

In this article, I'll explain compiler optimizations through a series of examples, focusing on what compilers do.

How Target-Independent is Your IR?

bookmark · Added on November 17, 2024 · 8 min read

An esoteric exploration on the target independence of compiler IRs.

Bibliopolis-Book-retypeset-1984

bookmark · Added on November 12, 2024 · 1h 34m read

Numerical Recipes

bookmark · Added on November 11, 2024 · 1 min read

We are Numerical Recipes, one of the oldest continuously operating sites on the Internet.

Unpacking Intuition

bookmark · Added on November 10, 2024 · 12 min read

Can intuition be taught? The way in which faces are recognized, the structure of natural classes, and the architecture of intuition may all be instances of the same process. The conjecture that intuition is a species of recognition memory implies ...

For Beginners

bookmark · Added on November 9, 2024 · 12 min read

Occasional writings about Haskell.

Oasis: A Universe in a Transformer

bookmark · Added on October 31, 2024 · 4 min read

Generating Worlds in Realtime

A Fat Pointer Library

bookmark · Added on October 27, 2024 · 8 min read

libCello Official Website

The Basics

bookmark · Added on October 27, 2024 · 2 min read

Here’s what I consider to be the basics.

TCP Server in Zig - Part 5a - Poll

bookmark · Added on October 15, 2024 · 24 min read

Using non-blocking sockets and poll to improve the scalability of our system.

6.824 Schedule: Spring 2022

bookmark · Added on October 1, 2024 · 1 min read

Here is the tentative schedule of lectures and due dates. The lecture notes and paper questions for future dates are copies from previous years, and may change.

2305.20091

bookmark · Added on September 30, 2024 · 48 min read

Humans in 4D: Reconstructing and Tracking Humans with Transformers

bookmark · Added on September 30, 2024 · 1 min read

Join the discussion on this paper page

slpj-book-1987.djvu

bookmark · Added on September 30, 2024 · 9h 42m read

Typing the technical interview

bookmark · Added on September 30, 2024 · 15 min read

In the formless days, long before the rise of the Church, all spells were woven of pure causality, all actions were permitted, and death was common.

Reversing the technical interview

bookmark · Added on September 30, 2024 · 3 min read

If you want to get a job as a software witch, you’re going to have to pass a whiteboard interview.

Hexing the technical interview

bookmark · Added on September 30, 2024 · 13 min read

But Hacker News has read of you, in their snicker-slithing susurrential warrens, and word has spread, which is why the young man offering you a smörgåsbord of microkitchen delights looks mildly suspicious already.

Nine Rules for SIMD Acceleration of Your Rust Code (Part 1)

bookmark · Added on September 23, 2024 · 20 min read

General Lessons from Boosting Data Ingestion in the range-set-blaze Crate by 7x

Conscious exotica

bookmark · Added on September 21, 2024 · 42 min read

From algorithms to aliens, could humans ever understand minds that are radically unlike our own?

B-trees and database indexes

bookmark · Added on September 13, 2024 · 22 min read

B-trees are used by many modern DBMSs. Learn how they work, how databases use them, and how your choice of primary key can affect index performance.

Safe C++

bookmark · Added on September 13, 2024 · 2h 13m read

Over the past two years, the United States Government has been issuing warnings about memory-unsafe programming languages with increasing urgency.

Tutorial on Diffusion Models for Imaging and Vision

bookmark · Added on September 10, 2024 · 2h 48m read

The astonishing growth of generative tools in recent years has empowered many exciting applications in text-to-image generation and text-to-video generation. The underlying principle behind these generative tools is the concept of diffusion, a particular sampling mechanism that has overcome some shortcomings that were deemed difficult in the previous approaches. The goal of this tutorial is to discuss the essential ideas underlying the diffusion models. The target audience of this tutorial includes undergraduate and graduate students who are interested in doing research on diffusion models or applying these models to solve other problems.

Async Rust can be a pleasure to work with (without `Send + Sync + 'static`)

bookmark · Added on September 9, 2024 · 26 min read

Async Rust is powerful. And it can be a pain to work with (and learn). Async Rust can be a pleasure to work with, though, if we can do it without `Send + Sync + 'static`.

The Perfect Plan

bookmark · Added on September 3, 2024 · 12 min read

Too often do we obsess over the perfect plan to chase our dreams, resulting in analysis paralysis. Instead of being stuck in this limbo, I've made the perfect plan for anyone to chase their dreams.

The Fast Track

bookmark · Added on September 1, 2024 · 1 min read

In order to accelerate the development of prospective mathematical scientists, we have selected a series of textbooks one can study to reach expertise in mathematics and physics in the most efficient manner possible.

Zig's BoundedArray

bookmark · Added on August 27, 2024 · 4 min read

A quick look at how and why to use Zig's BoundedArray.

Linus Torvalds talks AI, Rust adoption, and why the Linux kernel is 'the only thing that matters'

bookmark · Added on August 24, 2024 · 5 min read

In a wide-ranging conversation with Verizon open-source officer Dirk Hohndel, 'plodding engineer' Linus Torvalds discussed where Linux is today and where it may go tomorrow.

Intercepting and modifying Linux system calls with ptrace

bookmark · Added on August 24, 2024 · 14 min read

Intercepting and modifying Linux system calls with ptrace

What's the big deal about Deterministic Simulation Testing?

bookmark · Added on August 24, 2024 · 14 min read

What's the big deal about Deterministic Simulation Testing?

Zig and Emulators

bookmark · Added on August 24, 2024 · 19 min read

Some quick Zig feedback in the context of a new 8-bit emulator project I starteda little while ago:

Linkers part 1

bookmark · Added on August 24, 2024 · 3 min read

I’ve been working on and off on a new linker.

A ToC of the 20 part linker essay

bookmark · Added on August 24, 2024 · 1 min read

I release this message (the ToC and comments) into the public domain, no right reserved.

trading_interview_blog

bookmark · Added on August 21, 2024 · 25 min read

`zig cc`: a Powerful Drop-In Replacement for GCC/Clang

bookmark · Added on August 15, 2024 · 24 min read

If you have heard of Zig before, you may know it as a promising new programming language which is ambitiously trying to overthrow C as the de-facto systems language.

Zig Build System

bookmark · Added on August 15, 2024 · 27 min read

The fundamental commands zig build-exe, zig build-lib, zig build-obj, and zig test are often sufficient.

Resources for Amateur Compiler Writers

bookmark · Added on August 14, 2024 · 4 min read

I know complete pans of the literature are left out, but this is a page for amateur compiler writers. Anything that I did not find practical is not listed here.

MattPD/cpplinks: A categorized list of C++ resources.

bookmark · Added on August 14, 2024 · 1 min read

A categorized list of C++ resources. Contribute to MattPD/cpplinks development by creating an account on GitHub.

Putting the “You” in CPU

bookmark · Added on August 14, 2024 · 1 min read

Curious exactly what happens when you run a program on your computer? Learn how multiprocessing works, what system calls really are, how computers manage memory with hardware interrupts, and how Linux loads executables.

How to Compile Your Language

bookmark · Added on August 9, 2024 · 3 min read

The guide also covers how to create a platform-specific executable with the help of the LLVM compiler infrastructure, which all of the previously mentioned languages use for the same purpose.

Introduction to the Odin Programming Language

bookmark · Added on August 9, 2024 · 59 min read

Preface This article is an introduction the Odin Programming Language. It is aimed at people who know a bit of programming, but have never touched Odin. It is not a reference guide, rather I try to keep things informal and talk about what I think are important aspects of the language. There will be some notes on differences to C/C++, as Odin in many ways tries to be better C. If you enjoy this article and want to support me, then you can do so by becoming a patron.

Arena allocator tips and tricks

bookmark · Added on August 6, 2024 · 12 min read

Over the past year I’ve refined my approach to arena allocation. With practice, it’s effective, simple, and fast; typically as easy to use as garbage collection but without the costs.

Part 2: Portable Executable Files

bookmark · Added on August 6, 2024 · 24 min read

bytecode interpreters for tiny computers

bookmark · Added on August 4, 2024 · 48 min read

I've previously come to the conclusion that there's little reason for using bytecode in the modern world, except in order to get more compact code, for which it can be very effective.

How I built zig-sqlite

bookmark · Added on August 4, 2024 · 9 min read

When you prepare a statement zig-sqlite creates a brand new type only for this prepared statement.

The Hunt for the Missing Data Type

bookmark · Added on August 3, 2024 · 14 min read

A (directed) graph is a set of nodes, connected by arrows (edges). The nodes and edges may contain data. Here are some graphs: All graphs made with graphviz (source) Graphs are ubiquitous in software engineering: Package dependencies form directed graphs, as do module imports. The internet is a graph of links between webpages. Model checkers analyze software by exploring the “state space” of all possible configurations.

Microfeatures I'd like to see in more languages

bookmark · Added on August 3, 2024 · 6 min read

There are roughly three classes of language features: Features that the language is effectively designed around, such that you can't add it after the fact....

Google’s Fully Homomorphic Encryption Compiler — A Primer

bookmark · Added on August 2, 2024 · 17 min read

Back in May of 2022 I transferred teams at Google to work on Fully Homomorphic Encryption (newsletter announcement). Since then I’ve been working on a variety of projects in the space, includ…

Will I be able to access proprietary platform APIs (e.g. Android / iOS)?

bookmark · Added on August 1, 2024 · 13 min read

The kind of binary format being considered for WebAssembly can be natively decoded much faster than JavaScript can be parsed (experiments show more than 20× faster).

The future of Clang-based tooling

bookmark · Added on August 1, 2024 · 11 min read

By Peter Goodman Clang is a marvelous compiler; it’s a compiler’s compiler! But it isn’t a toolsmith’s compiler. As a toolsmith, my ideal compiler would be an open book, allowing me to get to…

Fast Multidimensional Matrix Multiplication on CPU from Scratch

bookmark · Added on July 30, 2024 · 15 min read

Numpy can multiply two 1024x1024 matrices on a 4-core Intel CPU in ~8ms.This is incredibly fast, considering this boils down to 18 FLOPs / core / cycle, with...

Efficient n-states on x86 systems

bookmark · Added on July 29, 2024 · 4 min read

Program tuning as a resource allocation problem

bookmark · Added on July 29, 2024 · 6 min read

How web bloat impacts users with slow connections

bookmark · Added on July 29, 2024 · 59 min read

Files are hard

bookmark · Added on July 29, 2024 · 24 min read

Ringing in a new asynchronous I/O API

bookmark · Added on July 29, 2024 · 9 min read

applicative-mental-models

bookmark · Added on July 29, 2024 · 6 min read

Optimizing subroutines in assembly language

bookmark · Added on July 29, 2024 · 5h 19m read

Demystifying programs that create programs, part 1: A disassembler

bookmark · Added on July 29, 2024 · 16 min read

QBE vs LLVM

bookmark · Added on July 29, 2024 · 2 min read

Recent presentations and papers

bookmark · Added on July 29, 2024 · 9 min read

brotli-2015-09-22

bookmark · Added on July 29, 2024 · 6 min read

How long does it take to make a context switch?

bookmark · Added on July 29, 2024 · 14 min read

Ghostty Devlog 001

bookmark · Added on July 29, 2024 · 7 min read

Tiled Matrix Multiplication

bookmark · Added on July 29, 2024 · 5 min read

Rust Atomics and Locks

bookmark · Added on July 29, 2024 · 3 min read

Compiler Backend

bookmark · Added on July 29, 2024 · 3 min read

Vale's Memory Safety Strategy: Generational References and Regions

bookmark · Added on July 29, 2024 · 19 min read

Introduction

bookmark · Added on July 29, 2024 · 4 min read

Cache-Oblivious Algorithms

bookmark · Added on July 29, 2024 · 4 min read

A Memory Allocator

bookmark · Added on July 29, 2024 · 13 min read

Cramming: Training a Language Model on a Single GPU in One Day

bookmark · Added on July 29, 2024 · 1h 2m read

The MiniPile Challenge for Data-Efficient Language Models

bookmark · Added on July 29, 2024 · 18 min read

Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget

bookmark · Added on July 29, 2024 · 1 min read

1024cores

bookmark · Added on July 29, 2024 · 1 min read

Implementing interactive languages

bookmark · Added on July 28, 2024 · 8 min read

Pointers Are Complicated, or: What's in a Byte?

bookmark · Added on July 28, 2024 · 14 min read

Three Architectures for a Responsive IDE

bookmark · Added on July 28, 2024 · 10 min read

How a Zig IDE Could Work Feb 10, 2023

bookmark · Added on July 28, 2024 · 9 min read

Properly Testing Concurrent Data Structures Jul 5, 2024

bookmark · Added on July 28, 2024 · 23 min read

Parse, don’t validate

bookmark · Added on July 28, 2024 · 20 min read

Too Fast, Too Megamorphic: what influences method call performance in Java?

bookmark · Added on July 28, 2024 · 11 min read

The Black Magic of (Java) Method Dispatch

bookmark · Added on July 28, 2024 · 1 min read

Why null sucks, even if it's checked

bookmark · Added on July 28, 2024 · 14 min read

Unnamed Document

bookmark · Added on July 28, 2024 · 1 min read

Resources for Building Programming Languages

bookmark · Added on July 28, 2024 · 4 min read

Little 'Big Ideas' in Programming Language Design

bookmark · Added on July 28, 2024 · 9 min read

Computer Networking: A Top-Down Approach

bookmark · Added on July 27, 2024 · 2 min read

Using Uninitialized Memory for Fun and Profit Posted on Friday, March 14, 2008.

bookmark · Added on July 27, 2024 · 5 min read

Zip Files All The Way Down

bookmark · Added on July 27, 2024 · 12 min read

UTF-8: Bits, Bytes, and Benefits Posted on Friday, March 5, 2010.

bookmark · Added on July 27, 2024 · 4 min read

Minimal Boolean Formulas

bookmark · Added on July 27, 2024 · 25 min read

Hacking the OS X Kernel for Fun and Profiles Posted on Tuesday, August 13, 2013.

bookmark · Added on July 27, 2024 · 9 min read

How To Build a User-Level CPU Profiler Posted on Thursday, August 8, 2013.

bookmark · Added on July 27, 2024 · 9 min read

An Encoded Tree Traversal

bookmark · Added on July 27, 2024 · 5 min read

Our Software Dependency Problem

bookmark · Added on July 27, 2024 · 26 min read

The Magic of Sampling, and its Limitations Posted on Saturday, February 4, 2023.

bookmark · Added on July 27, 2024 · 9 min read

Running the “Reflections on Trusting Trust” Compiler Posted on Wednesday, October 25, 2023.

bookmark · Added on July 27, 2024 · 24 min read

Improving the Font Pipeline

bookmark · Added on July 26, 2024 · 22 min read

Easy Scalable Text Rendering on the GPU

bookmark · Added on July 26, 2024 · 6 min read

Adventures in Text Rendering: Kerning and Glyph Atlases

bookmark · Added on July 26, 2024 · 18 min read

Exploring the Power of Negative Space Programming

bookmark · Added on July 25, 2024 · 5 min read

CompilerTalkFinal

bookmark · Added on July 25, 2024 · 19 min read

Graydon Hoare: 21 compilers and 3 orders of magnitude in 60 minutes

bookmark · Added on July 25, 2024 · 1 min read

p75-hoare

bookmark · Added on July 25, 2024 · 41 min read

Updating the Go Memory Model

bookmark · Added on July 23, 2024 · 20 min read

Programming Language Memory Models (Memory Models, Part 2) Posted on Tuesday, July 6, 2021. PDF

bookmark · Added on July 23, 2024 · 45 min read

Hardware Memory Models (Memory Models, Part 1) Posted on Tuesday, June 29, 2021. PDF

bookmark · Added on July 23, 2024 · 27 min read

Baby Steps to a C Compiler

bookmark · Added on July 23, 2024 · 3 min read

Kernel Programming Guide

bookmark · Added on July 23, 2024 · 1 min read

Essential information for programming in the OS X kernel. Includes a high-level overview.

Tiny Tapeout

bookmark · Added on July 23, 2024 · 1 min read

Why Pascal is Not My Favorite Programming Language

bookmark · Added on July 23, 2024 · 31 min read

What Color is Your Function?

bookmark · Added on July 23, 2024 · 16 min read

What is an Invariant? Oct 6, 2023

bookmark · Added on July 22, 2024 · 9 min read

Chess-GPT's Internal World Model

bookmark · Added on July 22, 2024 · 17 min read

Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models

bookmark · Added on July 22, 2024 · 1 min read

Manipulating Chess-GPT's World Model

bookmark · Added on July 22, 2024 · 15 min read

Crafting an Interpreter in Zig - part 1

bookmark · Added on July 22, 2024 · 3 min read

Teach Yourself Programming in Ten Years

bookmark · Added on July 21, 2024 · 13 min read

What Every Computer Scientist Should Know About Floating-Point Arithmetic

bookmark · Added on July 21, 2024 · 2h 45m read

The Development of the C Language*

bookmark · Added on July 20, 2024 · 43 min read

Class Warfare

bookmark · Added on July 20, 2024 · 5 min read

Ownership

bookmark · Added on July 20, 2024 · 1 min read

A Note About Zig Books for the Zig Community

bookmark · Added on July 18, 2024 · 8 min read

Your Starting Point!

bookmark · Added on July 17, 2024 · 25 min read

Zig Interfaces for the Uninitiated, an update

bookmark · Added on July 17, 2024 · 6 min read

Zig Interfaces for the Uninitiated

bookmark · Added on July 17, 2024 · 8 min read

Exploring Compile-Time Interfaces in Zig

bookmark · Added on July 17, 2024 · 7 min read

Aro - a C compiler

bookmark · Added on July 17, 2024 · 1 min read

Database Systems

bookmark · Added on July 17, 2024 · 1 min read

Discovering and exploring mmap using Go

bookmark · Added on July 17, 2024 · 7 min read

But how, exactly, databases use mmap?

bookmark · Added on July 17, 2024 · 10 min read

reHow memory mapped files, filesystems and cloud storage works

bookmark · Added on July 17, 2024 · 2 min read

Implementing a file pager in Zig

bookmark · Added on July 17, 2024 · 5 min read

Criticizing Hare language approach for generic data structures

bookmark · Added on July 17, 2024 · 5 min read

spikedoanz/from-bits-to-intelligence: machine learninig stack in under 100,000 lines of code

bookmark · Added on July 16, 2024 · 2 min read

One year of C

bookmark · Added on July 16, 2024 · 8 min read

Heap Memory and Allocators

bookmark · Added on July 15, 2024 · 22 min read

Pointers

bookmark · Added on July 15, 2024 · 16 min read

Emulator 101

bookmark · Added on July 15, 2024 · 1 min read

A detailed, step by step guide to writing an emulator

Data Compression Explained

bookmark · Added on July 15, 2024 · 4h 4m read

Twitter's Recommendation Algorithm

bookmark · Added on July 14, 2024 · 7 min read

Programming languages resources

bookmark · Added on July 13, 2024 · 3 min read

3D Math Primer for Graphics and Game Development

bookmark · Added on July 12, 2024 · 1 min read

Welcome to OpenGL

bookmark · Added on July 12, 2024 · 3 min read

WebGPU Fundamentals

bookmark · Added on July 12, 2024 · 1 min read

An opinionated beginner’s guide to Haskell in mid-2019

bookmark · Added on July 12, 2024 · 17 min read

Are tagged unions overrated?

bookmark · Added on July 12, 2024 · 11 min read

C++ Core Guidelines

bookmark · Added on July 12, 2024 · 9h 40m read

What every systems programmer should know about concurrency

bookmark · Added on July 11, 2024 · 32 min read

compiler_construction

bookmark · Added on July 11, 2024 · 47 min read

How do we tell truths that might hurt?

bookmark · Added on July 11, 2024 · 3 min read

The next fifty years

bookmark · Added on July 11, 2024 · 10 min read

Recommender Systems: A Primer

bookmark · Added on July 10, 2024 · 2h 28m read

http client in the standard library · Issue #2007 · ziglang/zig

bookmark · Added on July 10, 2024 · 9 min read

Introduction to Compilers and Language Design

bookmark · Added on July 10, 2024 · 1 min read

Bare Metal Zig

bookmark · Added on July 10, 2024 · 10 min read

Comparing SIMD on x86-64 and arm64

bookmark · Added on July 10, 2024 · 1h 9m read

Compiler Optimizations Are Hard Because They Forget

bookmark · Added on July 10, 2024 · 11 min read

C Isn't A Programming Language Anymore

bookmark · Added on July 10, 2024 · 19 min read

Writing a C Compiler, Part 1

bookmark · Added on July 9, 2024 · 18 min read

GitHub - DoctorWkt/acwj: A Compiler Writing Journey

bookmark · Added on July 9, 2024 · 2 min read

A new JIT engine for PHP-8.4/9

bookmark · Added on July 9, 2024 · 1 min read

Unknown

bookmark · Added on July 9, 2024 · 1h 30m read

Introduction 2016 NUMA Deep Dive Series

bookmark · Added on July 9, 2024 · 3 min read

von Neumann architecture - Wikipedia

bookmark · Added on July 9, 2024 · 20 min read

Compiling tree transforms to operate on packed representations

bookmark · Added on July 8, 2024 · 1 min read

Pipelines Support Vectorized, Point-Free, and Imperative Style

bookmark · Added on July 8, 2024 · 4 min read

Entering text in the terminal is complicated

bookmark · Added on July 8, 2024 · 9 min read

What happens when you start a process on Linux?

bookmark · Added on July 8, 2024 · 5 min read

Debug your programs like they're closed source!

bookmark · Added on July 8, 2024 · 7 min read

How I got better at debugging

bookmark · Added on July 8, 2024 · 4 min read

Media Page Under Construction

bookmark · Added on July 8, 2024 · 1 min read

Infographics: Operation Costs in CPU Clock Cycles

bookmark · Added on July 8, 2024 · 23 min read

Handles are the better pointers

bookmark · Added on July 8, 2024 · 14 min read

You're Not Sick of Programming

bookmark · Added on July 8, 2024 · 1 min read

Zig Bare Metal Programming on STM32F103 — Booting up

bookmark · Added on July 8, 2024 · 15 min read

OWASP Top Ten

bookmark · Added on July 7, 2024 · 3 min read

Introduction

bookmark · Added on July 7, 2024 · 1 min read

The Copenhagen Book

bookmark · Added on July 7, 2024 · 1 min read

Undefined Behavior deserves a better reputation

bookmark · Added on July 6, 2024 · 12 min read

KHM+15

bookmark · Added on July 6, 2024 · 52 min read

Learning LLVM (Part-1) - Writing a simple LLVM pass

bookmark · Added on July 5, 2024 · 2 min read

Some Were Meant for C

bookmark · Added on July 5, 2024 · 1h 11m read

Xv6, a simple Unix-like teaching operating system

bookmark · Added on July 5, 2024 · 3 min read

C Is Not a Low-level Language

bookmark · Added on July 5, 2024 · 17 min read

Should you learn C to "learn how the computer works"?

bookmark · Added on July 5, 2024 · 12 min read

A Guide to Undefined Behavior in C and C++, Part 1

bookmark · Added on July 5, 2024 · 17 min read

Using neural nets to recognize handwritten digits

bookmark · Added on July 5, 2024 · 1h 11m read

When Network is Faster than Cache

bookmark · Added on July 5, 2024 · 12 min read

John Carmack on Functional Programming in C++

bookmark · Added on July 5, 2024 · 12 min read

Zig-style generics are not well-suited for most languages

bookmark · Added on July 5, 2024 · 12 min read

WebGL2 vs WebGL1

bookmark · Added on July 4, 2024 · 19 min read

WebGL How It Works

bookmark · Added on July 4, 2024 · 10 min read

The_Night_Watch

bookmark · Added on July 4, 2024 · 13 min read

FreeType

bookmark · Added on July 4, 2024 · 1 min read

A Freestanding Rust Binary

bookmark · Added on July 3, 2024 · 18 min read

Manually linking Rust binaries to support out-of-tree LLVM passes

bookmark · Added on July 3, 2024 · 7 min read

The Rust Reference

bookmark · Added on July 3, 2024 · 7 min read

Rust Compiler Development Guide

bookmark · Added on July 3, 2024 · 14 min read

How to speed up the Rust compiler one last time

bookmark · Added on July 3, 2024 · 9 min read

How to speed up the Rust compiler in March 2024

bookmark · Added on July 3, 2024 · 4 min read

Zig Bits 0x4: Building an HTTP client/server from scratch

bookmark · Added on July 3, 2024 · 19 min read

Do We Really Need A Link Step?

bookmark · Added on July 3, 2024 · 2 min read

Death Note: L, Anonymity & Eluding Entropy

bookmark · Added on July 2, 2024 · 52 min read

jamiebuilds/the-super-tiny-compiler: :snowman: Possibly the smallest compiler ever

bookmark · Added on July 2, 2024 · 1 min read

5 Days to Virtualization: A Series on Hypervisor Development

bookmark · Added on July 2, 2024 · 5 min read

In-depth analysis on Valorant’s Guarded Regions

bookmark · Added on July 2, 2024 · 14 min read

Exploit Development: No Code Execution? No Problem! Living The Age of VBS, HVCI, and Kernel CFG

bookmark · Added on July 2, 2024 · 1h 31m read

Reader

bookmark · Added on July 2, 2024 · 6 min read

CheerpX versus WebContainers

bookmark · Added on July 2, 2024 · 2 min read

Creating a Rootkit to Learn C

bookmark · Added on July 2, 2024 · 32 min read

Picsart-AI-Research/LIVE-Layerwise-Image-Vectorization: [CVPR 2022 Oral] Towards Layer-wise Image Vectorization

bookmark · Added on July 1, 2024 · 1 min read

Udacity CS344: Intro to Parallel Programming

bookmark · Added on July 1, 2024 · 2 min read

CS 361: Systems Programming

bookmark · Added on July 1, 2024 · 1 min read

Resolving Rust Symbols

bookmark · Added on July 1, 2024 · 15 min read

When FFI Function Calls Beat Native C

bookmark · Added on July 1, 2024 · 8 min read

Cap'n Proto, FlatBuffers, and SBE

bookmark · Added on July 1, 2024 · 14 min read

A Database Without Dynamic Memory Allocation

bookmark · Added on July 1, 2024 · 5 min read

Wizard Zines Collection!

bookmark · Added on July 1, 2024 · 1 min read

Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0

bookmark · Added on July 1, 2024 · 15 min read

Problems of C, and how Zig addresses them

bookmark · Added on July 1, 2024 · 13 min read

How to use hash map contexts to save memory when doing a string table

bookmark · Added on June 30, 2024 · 2 min read

resume.txt

bookmark · Added on June 30, 2024 · 9 min read

Leslie Lamport

bookmark · Added on June 28, 2024 · 3h 51m read

Indices and tables

bookmark · Added on June 27, 2024 · 1 min read

448997590_1496256481254967_2304975057370160015_n

bookmark · Added on June 27, 2024 · 1h 0m read

Bare Bones

bookmark · Added on June 25, 2024 · 27 min read

The Graphics Codex

bookmark · Added on June 24, 2024 · 1 min read

[2305.13009] Textually Pretrained Speech Language Models

bookmark · Added on June 24, 2024 · 1 min read

Notes on partial borrows

bookmark · Added on June 24, 2024 · 6 min read

Dioxus Labs + “High-level Rust”

bookmark · Added on June 24, 2024 · 23 min read

Compile-Time Configuration For Zig Libraries

bookmark · Added on June 24, 2024 · 2 min read

Generics

bookmark · Added on June 24, 2024 · 5 min read

Zig's HashMap - Part 1

bookmark · Added on June 24, 2024 · 12 min read

Zig Parser

bookmark · Added on June 24, 2024 · 15 min read

Copying Better: How To Acquire The Tacit Knowledge of Experts

bookmark · Added on June 24, 2024 · 29 min read

Causal ordering

bookmark · Added on June 24, 2024 · 4 min read

Assorted thoughts on zig (and rust)

bookmark · Added on June 24, 2024 · 24 min read

Columnar kernels in go?

bookmark · Added on June 24, 2024 · 11 min read

Over the winter I'm going to be adding a columnar query engine to an existing system written in go.

An opinionated map of incremental and streaming systems

bookmark · Added on June 24, 2024 · 7 min read

Internal consistency in streaming systems

bookmark · Added on June 24, 2024 · 32 min read

Pain we forgot

bookmark · Added on June 24, 2024 · 15 min read

Have you tried rubbing a database on it?

bookmark · Added on June 24, 2024 · 1 min read

The shape of data

bookmark · Added on June 24, 2024 · 29 min read

Reflections on a decade of coding

bookmark · Added on June 24, 2024 · 2 min read

Prospecting for Hash Functions

bookmark · Added on June 24, 2024 · 11 min read

The Missing Zig Polymorphism / Runtime Dispatch Reference

bookmark · Added on June 24, 2024 · 6 min read

Nanosystems

bookmark · Added on June 23, 2024 · 1 min read

How To Become A Hacker

bookmark · Added on June 23, 2024 · 46 min read

the rr debugging experience

bookmark · Added on June 20, 2024 · 7 min read

Text Buffer Reimplementation

bookmark · Added on June 19, 2024 · 15 min read

What Is The Minimal Set Of Optimizations Needed For Zero-Cost Abstraction?

bookmark · Added on June 19, 2024 · 6 min read

Using ASCII waveforms to test hardware designs

bookmark · Added on June 19, 2024 · 5 min read

Rust 2019 and beyond: limits to (some) growth.

bookmark · Added on June 19, 2024 · 10 min read

Your ABI is Probably Wrong

bookmark · Added on June 19, 2024 · 4 min read

GitHub - sirupsen/napkin-math: Techniques and numbers for estimating system's performance from first-principles

bookmark · Added on June 19, 2024 · 8 min read

Don't write bugs

bookmark · Added on June 19, 2024 · 1 min read

technicalities: "not rocket science" (the story of monotone and bors)

bookmark · Added on June 19, 2024 · 5 min read

Why is Python slow

bookmark · Added on June 19, 2024 · 5 min read

Design duality and the expression problem

bookmark · Added on June 19, 2024 · 13 min read

Random Thoughts On Rust: crates.io And IDEs

bookmark · Added on June 19, 2024 · 4 min read

John Carmack on Inlined Code

bookmark · Added on June 19, 2024 · 13 min read

A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World

bookmark · Added on June 19, 2024 · 34 min read

What is Systems Programming, Really?

bookmark · Added on June 19, 2024 · 12 min read

MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding

bookmark · Added on June 18, 2024 · 1 min read

Mitchell Hashimoto

bookmark · Added on June 17, 2024 · 1 min read

Understanding_Machine_Learning_-_From_Theory_to_Algorithms

bookmark · Added on June 17, 2024 · 12h 46m read

I'm sorry, but there is no content provided for me to summarize. If you provide me with the specific content or information you would like summarized, I would be happy to help.

UB Might Be a Wrong Term for Newer Languages Apr 2, 2023

bookmark · Added on June 17, 2024 · 1 min read

What Every C Programmer Should Know About Undefined Behavior #1/3

bookmark · Added on June 17, 2024 · 9 min read

The Rustonomicon

bookmark · Added on June 16, 2024 · 2 min read

chrono-Compatible Low-Level Date Algorithms

bookmark · Added on June 15, 2024 · 43 min read

Step-by-Step Diffusion: An Elementary Tutorial

bookmark · Added on June 15, 2024 · 1h 26m read

The text is a tutorial about diffusion. The authors are Preetum Nakkiran, Arwen Bradley, Hattie Zhou, and Madhu Advani. The tutorial is available on the domain readwise.io.

So Many New Systems Programming Languages II

bookmark · Added on June 14, 2024 · 12 min read

zackoverflow

bookmark · Added on June 14, 2024 · 1 min read

From Theory To Implementation

bookmark · Added on June 14, 2024 · 1 min read

Speech-to-text models

bookmark · Added on June 13, 2024 · 17 min read

Ray Tracing in One Weekend

bookmark · Added on June 13, 2024 · 1h 3m read

Untangling Lifetimes: The Arena Allocator

bookmark · Added on June 13, 2024 · 40 min read

Tree-Structured Concurrency — 2023-07-01

bookmark · Added on June 12, 2024 · 15 min read

immersivemath: Immersive Linear Algebra

bookmark · Added on June 12, 2024 · 1 min read

BSTJ 57: 6. July-August 1978: The UNIX Time-Sharing System. (Ritchie, D.M.; Thompson, K.)

bookmark · Added on June 12, 2024 · 4 min read

Principles of compiler design

bookmark · Added on June 12, 2024 · 1 min read

A Mathematical Theory of Communication

bookmark · Added on June 12, 2024 · 2h 24m read

Mapping the whole internet with Hilbert curves

bookmark · Added on June 12, 2024 · 4 min read

Hausdorff dimension - Wikipedia

bookmark · Added on June 12, 2024 · 1 min read

xorvoid

bookmark · Added on June 12, 2024 · 3 min read

A Recipe for Training Neural Networks

bookmark · Added on June 11, 2024 · 19 min read

You own your data, in spite of the cloud

bookmark · Added on June 11, 2024 · 54 min read

Writing CUDA Kernels for PyTorch

bookmark · Added on June 11, 2024 · 34 min read

Multi-Query & Grouped-Query Attention

bookmark · Added on June 11, 2024 · 13 min read

999 crates of Rust on the wall

bookmark · Added on June 11, 2024 · 9 min read

Uiuisms

bookmark · Added on June 9, 2024 · 1 min read

Arithmetic functions

bookmark · Added on June 9, 2024 · 7 min read

An interactive study of queueing strategies

bookmark · Added on June 9, 2024 · 11 min read

A DSL for Implementing Math Functions

bookmark · Added on June 8, 2024 · 1 min read

ethereumbook/04keys-addresses.asciidoc at develop · ethereumbook/ethereumbook · GitHub

bookmark · Added on June 6, 2024 · 34 min read

Accidentally Turing-Complete

bookmark · Added on June 6, 2024 · 8 min read

The Art of Computer Programming, Vol. 4 Fascicle 6

bookmark · Added on June 6, 2024 · 16h 13m read

The_Manga_Guide_to_Linear_Algebra

bookmark · Added on June 6, 2024 · 1h 47m read

Exploring architectures- Transformers II

bookmark · Added on June 6, 2024 · 6 min read

What are Diffusion Models?

bookmark · Added on June 6, 2024 · 23 min read

Problems with BQN

bookmark · Added on June 6, 2024 · 30 min read

Iterative α-(de)Blending: a Minimalist Deterministic Diffusion Model

bookmark · Added on June 5, 2024 · 1 min read

The borrow checker within

bookmark · Added on June 5, 2024 · 15 min read

How should I read type system notation?

bookmark · Added on June 5, 2024 · 11 min read

Writing a Simple Garbage Collector in C

bookmark · Added on June 5, 2024 · 19 min read

A decade of developing a programming language

bookmark · Added on June 5, 2024 · 14 min read

The Rust I Wanted Had No Future

bookmark · Added on June 5, 2024 · 15 min read

The Garbage Collection Handbook

bookmark · Added on June 5, 2024 · 2 min read

A high-bias, low-variance introduction to Machine Learning for physicists

bookmark · Added on June 5, 2024 · 6h 46m read

How diffusion models work: the math from scratch

bookmark · Added on June 1, 2024 · 18 min read

essentials-of-compilation

bookmark · Added on May 31, 2024 · 2h 54m read

PRACTICAL COMPILER CONSTRUCTION

bookmark · Added on May 31, 2024 · 1 min read

A Distributed Systems Reading List

bookmark · Added on May 31, 2024 · 4 min read

An Introduction to Assembly Programming with RISC-V

bookmark · Added on May 28, 2024 · 26 min read

This text provides information about a resource related to RISC-V programming. The ISBN number for this resource is 978-65-00-15811-3. It is authored by riscv-programming.org.

Microsoft PowerPoint - SRAM Architecture

bookmark · Added on May 28, 2024 · 4 min read

MLIR: A Compiler Infrastructure for the End of Moore's Law

bookmark · Added on May 27, 2024 · 59 min read

MLIR — Getting Started

bookmark · Added on May 27, 2024 · 13 min read

The text is a guide titled "MLIR — Getting Started" by Math ∩ Programming available on www.jeremykun.com.

Chapter 2 Basics of SIMD Programming

bookmark · Added on May 27, 2024 · 45 min read

Matrix multiplication in Mojo

bookmark · Added on May 27, 2024 · 10 min read

The text discusses matrix multiplication in Mojo. It is written by modular.com and can be found on docs.modular.com.

Matrix Multiplication on CPU

bookmark · Added on May 27, 2024 · 31 min read

The text is about matrix multiplication on a CPU. The author is Marek Kolodziej and the domain is marek.ai.

How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog

bookmark · Added on May 27, 2024 · 31 min read

The text is a worklog by Simon Boehm about optimizing a CUDA Matmul Kernel for cuBLAS-like performance. It can be found on the domain siboehm.com.

The Annotated Transformer

bookmark · Added on May 27, 2024 · 32 min read

Anonymity and the internet

bookmark · Added on May 27, 2024 · 3 min read

Auto-Regressive Next-Token Predictors are Universal Learners

bookmark · Added on May 26, 2024 · 40 min read

Where Vim Came From

bookmark · Added on May 25, 2024 · 14 min read

Building and operating a pretty big storage system called S3

bookmark · Added on May 25, 2024 · 31 min read

Unnamed Document

bookmark · Added on May 25, 2024 · 1 min read

LADW_2017-09-04

bookmark · Added on May 25, 2024 · 7h 55m read

New Scaling Laws for Large Language Models

bookmark · Added on May 25, 2024 · 7 min read

Binary Magic: Building BitNet 1.58bit Using PyTorch from Scratch

bookmark · Added on May 25, 2024 · 5 min read

king - man + woman is queen; but why?

bookmark · Added on May 25, 2024 · 9 min read

How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study

bookmark · Added on May 25, 2024 · 12 min read

1-bit Model

bookmark · Added on May 25, 2024 · 7 min read

Human Knowledge Compression Contest

bookmark · Added on May 25, 2024 · 48 min read

Heatmaps and CNNs Using Fast.ai

bookmark · Added on May 25, 2024 · 8 min read

Where do LLMs spend their FLOPS?

bookmark · Added on May 19, 2024 · 8 min read

The Annotated Diffusion Model

bookmark · Added on May 19, 2024 · 27 min read

Defusing Diffusion Models

bookmark · Added on May 19, 2024 · 21 min read

The Illustrated Stable Diffusion

bookmark · Added on May 19, 2024 · 11 min read

Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentation

bookmark · Added on May 10, 2024 · 16 min read

Sparse Autoencoders Find Highly Interpretable Features in Language Models

bookmark · Added on May 1, 2024 · 1 min read

KAN: Kolmogorov–Arnold Networks

bookmark · Added on May 1, 2024 · 1h 25m read

KAN: Kolmogorov-Arnold Networks

bookmark · Added on May 1, 2024 · 1h 43m read

Structure and Interpretation of Computer Programs, 2nd ed.

bookmark · Added on April 30, 2024 · 15h 41m read

OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework

bookmark · Added on April 25, 2024 · 26 min read

þÿAn Infinitely Large Napkin

bookmark · Added on April 23, 2024 · 26h 50m read

The text is titled "An Infinitely Large Napkin" by Evan Chen. The author's work can be found on readwise.io.

Transformers Meet Small Datasets

bookmark · Added on April 10, 2024 · 1 min read

Root Mean Square Layer Normalization

bookmark · Added on April 9, 2024 · 32 min read

Terry A. Davis

bookmark · Added on April 7, 2024 · 16 min read

Pattern Recognition and Machine Learning

bookmark · Added on April 6, 2024 · 22h 37m read

Ludwig Wittgenstein: The Duty of Genius

bookmark · Added on April 6, 2024 · 20h 33m read

Generative Agents: Interactive Simulacra of Human Behavior

bookmark · Added on March 28, 2024 · 1h 35m read

Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks

bookmark · Added on March 11, 2024 · 5h 7m read

Revisiting Deep Learning as a Non-Equilibrium Process

bookmark · Added on March 8, 2024 · 8 min read

Dissipative Adaptation: The Origins of Life and Deep Learning

bookmark · Added on March 8, 2024 · 7 min read

A Gentle Introduction to LLVM IR

bookmark · Added on March 6, 2024 · 31 min read

Re: [Fis] A PROPOSAL ABOUT THE DEFINITION OF INFORMATION

bookmark · Added on March 6, 2024 · 13 min read

The Art of Embeddings: Transforming Text for Vector Databases (Part 2)

bookmark · Added on March 6, 2024 · 13 min read

Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

bookmark · Added on March 6, 2024 · 34 min read

þÿThe Little Book of Deep Learning

bookmark · Added on March 6, 2024 · 1h 36m read

I'm sorry, but there is no content provided to summarize. If you have any text or information you would like me to summarize, please provide it so I can assist you.

Information

bookmark · Added on March 6, 2024 · 14 min read

Sequence to Sequence Learning with Neural Networks

bookmark · Added on March 6, 2024 · 26 min read

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

bookmark · Added on March 6, 2024 · 16 min read

How to round to 2 decimals with Python? [duplicate]

bookmark · Added on March 5, 2024 · 2 min read

Rounding floats with f-string [duplicate]

bookmark · Added on March 5, 2024 · 1 min read

Using %-formatting, I can specify the number of decimal cases in a string: x = 3.14159265 print('pi = %0.2f' %x) This would give me: pi = 3.14 Is there any way of doing this using f-strings in ...

Latent Interfaces

bookmark · Added on March 3, 2024 · 5 min read

Hypercomputation

bookmark · Added on March 3, 2024 · 16 min read

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

bookmark · Added on February 28, 2024 · 1 min read

How Netflix Really Uses Java

bookmark · Added on February 27, 2024 · 39 min read

Scheduling Internals

bookmark · Added on February 27, 2024 · 25 min read

Glossary of Deep Learning: Word Embedding

bookmark · Added on February 26, 2024 · 6 min read

gemini_v1_5_report

bookmark · Added on February 18, 2024 · 1h 59m read

How to Use t-SNE Effectively

bookmark · Added on February 16, 2024 · 9 min read

Consciousness, Cognition and the Neuronal Cytoskeleton – A New Paradigm Needed in Neuroscience

bookmark · Added on February 14, 2024 · 58 min read

OpenMEA: Open-Source Microelectrode Array Platform for Bioelectronic Interfacing

bookmark · Added on February 14, 2024 · 50 min read

Landauer's principle

bookmark · Added on February 14, 2024 · 7 min read

Bremermann's limit

bookmark · Added on February 14, 2024 · 2 min read

Bekenstein bound

bookmark · Added on February 14, 2024 · 10 min read

numerical_recipes

bookmark · Added on February 14, 2024 · 39h 52m read

Temperature as Joules per Bit

bookmark · Added on February 14, 2024 · 21 min read

Deep Learning Course

bookmark · Added on February 10, 2024 · 5 min read

Memory in Plain Sight: A Survey of the Uncanny Resemblances between Diffusion Models and Associative Memories

bookmark · Added on February 9, 2024 · 57 min read

2309.10668

bookmark · Added on February 8, 2024 · 37 min read

Memory in Plain Sight: A Survey of the Uncanny Resemblances between Diffusion Models and Associative Memories

bookmark · Added on February 8, 2024 · 1 min read

tns

bookmark · Added on February 8, 2024 · 9h 29m read

A New Physics Theory of Life | Quanta Magazine

bookmark · Added on February 7, 2024 · 11 min read

K-Level Reasoning with Large Language Models

bookmark · Added on February 7, 2024 · 1 min read

Competitive Programmer's Handbook

bookmark · Added on February 5, 2024 · 5h 55m read

Writing an OS in Rust

bookmark · Added on February 3, 2024 · 4 min read

Ever wanted to make your own programming language or wondered how they are designed and built?

bookmark · Added on February 3, 2024 · 1 min read

GitHub - sst/demo-ai-app: Sample AI movies app built with ❍ Ion

bookmark · Added on January 31, 2024 · 2 min read

ThermodynamicComputing

bookmark · Added on January 31, 2024 · 1h 14m read

Measuring Faithfulness in Chain-of-Thought Reasoning

bookmark · Added on January 28, 2024 · 1 min read

ageron/handson-ml3: A series of Jupyter notebooks that walk you through the fundamentals of Machine Learning and Deep Learning in Python using Scikit-Learn, Keras and TensorFlow 2.

bookmark · Added on January 26, 2024 · 2 min read

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

bookmark · Added on January 23, 2024 · 1 min read

Turing-1951 Intelligent Machinery-a Heretical Theory

bookmark · Added on January 21, 2024 · 10 min read

Self-Rewarding Language Models

bookmark · Added on January 20, 2024 · 1 min read

Software Development Trends 2023/2024 - Vol. 2.

bookmark · Added on January 16, 2024 · 8 min read

Word2vec from Scratch

bookmark · Added on January 15, 2024 · 19 min read

MemGPT: Towards LLMs as Operating Systems

bookmark · Added on January 15, 2024 · 39 min read

Visual Guides to understand the basics of Large Language Models

bookmark · Added on January 14, 2024 · 1 min read

Understanding and Coding Self-Attention, Multi-Head Attention, Cross-Attention, and Causal-Attention in LLMs

bookmark · Added on January 14, 2024 · 26 min read

Thinking in Systems: International Bestseller: Donella H. Meadows, Diana Wright: 9781603580557: Amazon.com: Books

bookmark · Added on January 14, 2024 · 5 min read

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

bookmark · Added on January 12, 2024 · 2h 42m read

This project is about how to systematically persuade LLMs to jailbreak them.

bookmark · Added on January 10, 2024 · 6 min read

Pruning vs Quantization: Which is Better?

bookmark · Added on January 10, 2024 · 38 min read

mlx-examples/lora at main · ml-explore/mlx-examples · GitHub

bookmark · Added on January 10, 2024 · 4 min read

Mixtral of Experts

bookmark · Added on January 10, 2024 · 1 min read

Paper page - Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

bookmark · Added on January 10, 2024 · 1 min read

The content is a set of instructions on how to cite a specific URL (arxiv.org/abs/2401.01335) in three different types of README.md files, in order to create links from those pages.