Infrastructure & inference
153 concepts the autonomous brain knows in this domain.
Self-hosted GPU cluster for local inference.
High-throughput local LLM serving.
Keep high-volume easy calls on-device; escalate only the hard ones.
Coverage -> build -> deploy; the source of truth for correctness.
blog-logic must stay fully covered or CI fails.
Static analysis gate on every change.
Git-grep gate that blocks leaked credentials.
A model reviews every diff before it ships.
Failed CI triggers an automatic git revert.
brain.json, watchdog.json, activity.json — the honest signals.
Impact/ease/risk-scored improvements the idea engine maintains.
Reducing model weights to lower-precision integers so frontier LLMs fit and run on modest local hardware without a…
Kubernetes orchestration for AI/ML workloads — GPU scheduling, autoscaling, and inference serving at scale.
Anthropic's Claude Mythos — a frontier, cybersecurity-specialized model that finds zero-day vulnerabilities; not…
Packaging an app plus its dependencies into a portable image that runs identically anywhere; the unit most sandboxes…
Amazon Web Services — the largest public cloud: compute, storage, and managed AI/ML (e.g. Bedrock).
Microsoft's public cloud, with deep first-party AI integration (Azure OpenAI, AI Foundry).
Declaratively provisioning and versioning infrastructure (e.g. Terraform) so environments are reproducible and…
Operationalizing ML/LLM systems end to end — deployment, monitoring, evaluation, and retraining.
Authorized simulated attacks that probe a system for exploitable weaknesses before real adversaries find them.
Exploitable flaws in code — injection, memory-safety bugs, insecure deserialization, CVEs, and zero-days.
An inference acceleration method where a small draft model proposes tokens that the larger target model verifies in…
Splitting a model's serving across multiple nodes (via tensor/pipeline parallelism) so a cluster can run models too…
A sparse architecture that routes each token to a small subset of expert sub-networks, so a model with very large total…
Models with publicly released weights (e.g. the GLM family) that can be self-hosted and run locally rather than called…
Fitting a model too large for available accelerator memory by spilling layers or the KV cache to CPU RAM, disk, or a…
Owning the full inference stack — hardware, open weights, and harness — instead of renting cloud model APIs, for…
Serving optimization that stores and reuses the attention KV cache for shared token prefixes across requests, cutting…
A service mesh that adds traffic management, security (mTLS), and observability to microservices on Kubernetes.
Routing scraper and bot traffic through pools of consumer IP addresses to evade rate limits and bot detection.
An LLM-serving scheduler that adds and evicts sequences from a running GPU batch at the token level, keeping…
The unit-cost analysis of serving model inference (cost per token, GPU capex and utilization) that drives choices like…
Coherent CPU-GPU shared address space (as on Grace-Blackwell/GB10 nodes) that lets large models spill across host and…
A content-delivery-network edge layer that caches and serves static assets near users and is the real point where…
A build technique (Cosmopolitan Libc / Actually Portable Executable) that packs a service into one self-contained…
Capping the volume of requests a client can make over a time window, used as a volumetric throttle against abusive or…
A challenge-response test designed to distinguish human users from automated bots before granting access to a resource.
The coherent high-bandwidth link (NVLink / ConnectX-class) that binds multiple GPU nodes into one memory-and-compute…
Automated crawlers run to harvest public web content at scale for LLM training and retrieval, the adversary that…
A deliberately planted decoy resource — such as a fake contact email or trap URL — used to detect, trap, or waste the…
Feeding scrapers deliberately falsified or decoy data so the datasets they harvest are corrupted and lose value for…
Serving LLM inference on commodity CPUs instead of GPUs, trading peak throughput for lower cost and broader hardware…
The observed flattening of model-quality gains from ever-larger parameter counts and pretraining compute, which…
Blocking or throttling requests by the reputation of their source IP or ASN (datacenter vs residential, known-abuse…
A flow-control mechanism where an overloaded component signals upstream producers to slow down, preventing unbounded…
A middleware layer (e.g. PgBouncer) that multiplexes many client connections onto a small reusable set of backend…
An open, royalty-free instruction set architecture enabling fully open and auditable hardware for running AI workloads…
A model is served by pooling heterogeneous consumer/edge devices into a peer-to-peer swarm, unlike datacenter…
A peer-to-peer transport substrate (content-addressed, NAT-traversing, QUIC-based, e.g. the iroh stack) that lets nodes…
Training or fine-tuning a shared model across many decentralized nodes on their local data, exchanging model updates…
Inference servers that profile their live workload and self-tune execution (batching, caching, kernel and route…
Peer-to-peer transport that addresses data by cryptographic hash of its content rather than by host location, enabling…
The technique of establishing direct peer-to-peer connections between nodes behind NATs/firewalls (e.g. via QUIC hole…
An encrypted, multiplexed transport built on UDP that connects peer nodes for distributed inference and…
A networking scheme where a node's cryptographic public key IS its dialable address, so peers mutually authenticate…
A capital pattern where AI chip and cloud vendors invest in the very customers who buy their compute, inflating…
Cryptographic or statistical techniques that let a requester confirm an untrusted remote peer actually executed the…
A Rust peer-to-peer networking stack that meshes hosts with QUIC transport, content-addressed blobs, and NAT…
Splitting an individual model layer's weight matrices across multiple GPUs or nodes so a single forward pass executes…
LLM token generation is limited by how fast model weights stream from memory rather than by raw compute, so…
Epidemic anti-entropy dissemination in which peers randomly exchange state so membership and data converge across a…
A distributed-inference strategy that splits a model's layers into contiguous stages placed on different devices, so…
The per-token attention key/value tensors an LLM caches during decoding, which dominate inference memory and grow…
An inference architecture that runs the compute-bound prefill phase and the memory-bandwidth-bound decode phase on…
The short useful economic life and rapid obsolescence of AI GPUs, whose depreciation schedule dominates the true unit…
Pooling idle, donated, and untrusted consumer machines into a shared substrate for AI inference and training as an…
The all-reduce / all-gather / broadcast primitives that synchronize tensors across many accelerators, forming the…
Consensus protocols like Raft and Paxos let a cluster of unreliable peer nodes agree on a single ordered log of state…
Sharding horizontally partitions a dataset across nodes by key so each node holds only a subset, scaling storage and…
Replication maintains synchronized copies of data on multiple nodes so the system survives node loss and can serve…
Scheduling or shifting AI inference and training workloads to the times and regions where the electricity grid is…
The ratio of a datacenter's total energy draw to the energy actually delivered to its compute, the standard durable…
Running models and inference inside hardware-backed trusted execution environments so an untrusted host or peer node…
Efficiency gains in compute lower the effective price of inference and thereby increase total demand and energy use…
The greenhouse-gas cost of manufacturing and disposing of AI hardware (chip fab, board, datacenter build), which can…
A design principle where a system's power draw scales down with utilization so idle and lightly-loaded hardware wastes…
Water Usage Effectiveness (WUE) measures the liters of water a datacenter consumes per kilowatt-hour of IT load, the…
Behind-the-meter power co-locates AI datacenters directly with dedicated generation (nuclear SMRs, gas, renewables) to…
A datacenter that dynamically modulates or shifts its compute load in response to real-time electrical-grid demand and…
A financing structure where AI datacenter buildouts are funded by debt secured against the GPUs and compute-purchase…
Remote Direct Memory Access lets one node read or write another node's memory over the network without involving either…
A class of GPU-specialized cloud providers that build and rent out accelerator capacity outside the traditional…
A decentralized key-value lookup structure that lets peers discover and route to each other without a central…
Decentralized physical infrastructure networks use token/economic incentives to coordinate independently owned hardware…
A KV-cache memory-management technique pioneered by vLLM that stores attention keys and values in non-contiguous…
A serving optimization that splits a long prompt's prefill into smaller chunks interleaved with ongoing decode steps,…
A serving-latency metric measuring the delay from request arrival to the first generated token, dominated by prompt…
IEEE-754 transcendental and accumulation results differ subtly across CPU/GPU, libm, and compiler, so the same…
Combining multiple GPU operations into a single custom kernel (e.g. FlashAttention-style attention) to cut memory…
Automatically searching over and generating GPU kernel variants to select the fastest for a given operation shape and…
Caching LLM responses keyed by embedding similarity so semantically equivalent queries return a stored answer instead…
Mirroring live production traffic to a candidate model so its quality, latency, and cost can be measured against the…
An attention technique that computes only a selected subset of query-key interactions instead of the full quadratic…
An IO-aware exact-attention algorithm that tiles the computation and avoids materializing the full attention matrix in…
A model-parallelism strategy that shards the sequence/context dimension across multiple devices so attention over…
The minimal set of hardware, firmware, and software whose correctness is critical to a system's security, such that a…
Transformer models dump excess attention onto a few initial tokens, so those tokens' KV-cache entries must be retained…
An attack that recovers secrets by observing indirect microarchitectural or physical effects — cache timing,…
A distributed attention algorithm that shards a long sequence across devices and overlaps blockwise attention with…
Attention variants that replace the quadratic softmax with kernel/feature maps or recurrent state to reach near-linear…
An attention variant that shares key/value projections across groups of query heads to shrink the KV cache and ease the…
A long-context technique that restricts each token's attention to a fixed local window, giving linear cost over…
Compressing or quantizing the transformer KV cache (per-token, per-channel, or via eviction) to cut the…
An attention variant that compresses keys and values into a low-rank latent vector, shrinking the KV cache read each…
A hardware-aligned, end-to-end trainable sparse-attention design that learns which key/value blocks to attend to,…
Route a query through a chain of increasingly capable and costly models, stopping at the first whose output passes a…
A parallelism strategy that shards a Mixture-of-Experts model's experts across devices so each accelerator hosts a…
Load-balancing that steers a request to the inference replica already holding the largest matching prefix in its KV…
Training a smaller student model to imitate a larger teacher model's outputs so it reaches near-teacher quality at far…
Adapting a frozen base model by training only small added weight matrices (e.g. LoRA adapters) rather than all…
Operating software on infrastructure you own and control rather than renting a managed SaaS, favoring small…
Computing hardware whose schematics, board layouts, or chip designs are released under open licenses so anyone can…
A lightweight hardware-virtualized sandbox (e.g. Firecracker-style) that boots in milliseconds to give an agent…
A disposable, provisioned-per-task compute environment that is created fresh for an agent and torn down afterward, so…
The per-token generation delay (time-per-output-token) during decode, distinct from prefill's time-to-first-token, and…
The rate of LLM requests completed within their latency SLO (not raw throughput), the metric a team actually optimizes…
A performance model that bounds a kernel's achievable throughput by either peak compute or memory bandwidth given its…
A confined agent or workload breaching its isolation boundary (VM, container, or microVM) to reach the host or other…
Restricting which kernel system calls a process may invoke (e.g. seccomp-BPF or Landlock) to shrink the kernel attack…
A runtime exploit mitigation that constrains a program's indirect jumps and calls to valid targets, blocking…
A user-space kernel (e.g. gVisor-style) that intercepts and re-implements a guest workload's syscalls so an untrusted…
A single-address-space image that compiles an application with only the OS library functions it needs into a minimal,…
An operations paradigm where servers and environments are never modified in place but are torn down and rebuilt…
A cache-coherent chip-to-chip or node-to-node link (e.g. NVLink-C2C in Grace-Blackwell, or CXL) that lets a CPU and…
Running AI locally through models and runtimes built into the operating system or hardware vendor rather than bundling…
Repurposing decommissioned, consumer-grade, or e-waste accelerators for AI inference to lower cost and amortize…
A design philosophy that maximizes the useful lifespan of computing hardware and minimizes energy and material waste by…
The practice of durably archiving and redundantly mirroring public datasets and content so they survive deletion, link…
Lots Of Copies Keep Stuff Safe — a preservation paradigm that keeps public data durable by mirroring it across many…
A government-mandated interception or decryption pathway built into communications infrastructure, which weakens…
The economic model in which firms extract behavioral and personal data as raw material for prediction and AI training,…
The organized community practice of mirroring and archiving at-risk institutional or public datasets before they can be…
A compact, few-billion-parameter model designed to run efficiently on-device or on modest local hardware rather than in…
Running ML models entirely client-side in the browser via WebAssembly/WebGPU, turning media into structured data with…
Automatic speech recognition that transcribes audio incrementally in real time, emitting partial and revisable…
An always-on, low-power on-device audio technique that continuously listens for a small set of trigger words (e.g. a…
Running machine-learning inference directly inside a database engine (e.g. executing neural networks as SQL queries) so…
Empirical power-law relationships predicting how model loss falls with training compute, dataset size, and parameter…
Optical interconnect fabricated on silicon that moves data between accelerators as light, relieving the bandwidth and…
Operating systems increasingly ship built-in, benchmarkable on-device model capabilities (speech, vision, language) as…
The practice of running machine-learning models on microcontrollers and other resource-constrained embedded substrates…
A remote, fully-provisioned, standardized coding workspace (devcontainers, Codespaces, throwaway VMs) that replaces the…
The paradigm of funding open AI models and compute as institutionally-backed public goods rather than leaving them…
A Linux asynchronous I/O interface using shared submission and completion ring buffers to batch and offload syscalls,…
Loading model weights via mmap so the OS page cache streams them from disk on demand, letting a machine run models…
A desktop-class machine with large unified memory and integrated accelerators that runs and fine-tunes sizeable models…
A standardized single-file container that stores model weights (often quantized) with metadata so they load safely,…
A registry that indexes, versions, and distributes openly released model weights and datasets so anyone can discover…
Representing model weights in ~1.58 bits (ternary {-1,0,1}) so matrix multiplies collapse to additions, letting large…
Serving a Mixture-of-Experts model on limited memory by keeping only the currently-active experts resident and…
Combining the weights of multiple separately fine-tuned models (e.g. via linear interpolation, SLERP, or task…
The empirical trend (the 'densing law') that the capability achievable per model parameter roughly doubles over time,…
Removing redundant individual weights or whole structural blocks from a trained network to shrink its stored size and…