- TensorFlow
- Hugging Face: blog📡, learn
- Kaggle
- BigML
- SentenceTransformers
- Lightning AI’s Deep Learning Fundamentals
- Pytorch
- Machine Learning for Software Engineering: a curated list of papers, PhD theses, datasets, and tools
Optimisation
: Homepage, Blog, Old blog, YouTube📡⇈
: YouTube📡
: YouTube📡
: Homepage, Ahead of AI📡↑
AI Coffee Break with Letitia (): YouTube📡, Substack
Courses
Articles and videos
- SDS 771: Gradient Boosting: XGBoost, LightGBM and CatBoost, with Kirill Eremenko↑ (⧉) by and (2 April 2024) ► A good basic presentation of some algorithm based on decision trees: XGBoost, LightGBM, and CatBoost.
- SDS 831: PyTorch Lightning Lit-Serve and Lightning Studios, with Dr. Luca Antiga (⧉) by and (29 October 2024) ► The products of Lightning AI (PyTorch Lightning, Lightning Studios, LitServe, and Lightning Thunder) and some thoughts about small language models.
- Neural networks
- Polyworld: Using Evolution to Design Artificial Intelligence by (8 November 2007) ► Using artificial life to optimise neural networks.
- Evolving Neural Networks to Play 2048 by (12 May 2014) ► The title says it all.
- Is Dr. Calvin in the Room? by (16 March 2017) ► Some ideas and some dubious simple back-of-the-envelope calculations about neural networks.
- How to generate text: using different decoding methods for language generation with Transformers by (1 March 2020) ► A presentation of the different methods to control the text generated by a model: greedy search, beam search, sampling, top-K sampling, and top-p (nucleus) sampling.
- The Neural Network, A Visual Introduction by (23 August 2020) ► The title says it all.
- Gradients are Not All You Need (Machine Learning Research Paper Explained) by (16 November 2021) ► A paper ("Gradients are Not All You Need") showing that gradient backpropagation does not work properly for some chaotic systems.
- Machine Learning 1: Tour d'horizon et le cas MuZero (feat Dalle2, PaLM) - Passe-science #47 by (4 June 2022) ► The latest results of the best AIs and how MuZero is trained.
- ↪Machine Learning 2: Architecture et Alphastar (Transformer, attention) - Passe-science #48 by (11 June 2022) ► A description of the architecture of AlphaStar and of transformers.
- TensorFlow in 100 Seconds by (3 August 2022) ► A very short example of using TensorFlow.
- The spelled-out intro to neural networks and backpropagation: building micrograd↑ by (17 August 2022) ► A good basic introduction to backpropagation with the code details.
- Understanding Encoder And Decoder LLMs by (17 June 2023) ► The title says it all.
- Create a Large Language Model from Scratch with Python – Tutorial↓ by (25 August 2023) ► This lengthy tutorial is not worth watching. Many parts lack preparation, some explanations are confusing, spends a significant amount of time explaining simple concepts while skipping complex ones… We get some understanding of how to implement an LLM, but this would easily be accomplished in a one-hour video.
- LLM Training: RLHF and Its Alternatives by (10 September 2023) ► As said in the title, a clear description of RLHF and its alternatives.
- What is LoRA? Low-Rank Adaptation for finetuning LLMs EXPLAINED by (18 September 2023) ► A presentation of LoRA.
- Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation) — Things I Learned From Hundreds of Experiments by (19 November 2023) ► Some experiments with LoRA.
- What is Q-Learning (back to basics) by (25 November 2023) ► The title says it all.
- Mixture of Experts Explained by , , , , , and (11 December 2023) ► A technical history of MoE models.
- Apple is doing the UNTHINKABLE!!! by (6 January 2024) ► Some information about Apple’s MLX Framework.
- LLaMA Pro: Progressive LLaMA with Block Expansion (Paper Explained) by (7 January 2024) ► Commenting on a paper ("LLaMA Pro: Progressive LLaMA with Block Expansion") which contains dubious claims about how the researchers improved LLaMA by duplicating some blocks.
- A Guide to Deeplearning4j (8 January 2024) ► A short presentation of Deeplearning4j.
- Sampling for Text Generation↑ by (16 January 2024) ► A clear overview on the methods used to sample or constrain the output of a generative AI.
- LLMs itself CAN create BETTER LLMs by (23 January 2024) ► A quick presentation of "Self-Rewarding Language Models": a dubious claim that a model can be improved by training and rewarding a new iteration of itself.
- Model Merging, Mixtures of Experts, and Towards Smaller LLMs by (3 February 2024) ► Weight Averaged Reward Models, Tuning Language Models by Proxy, Mixtral of Experts, and TinyLlama.
- Sparse LLMs at inference: 6x faster transformers! | DEJAVU paper explained by (3 February 2024) ► A presentation of "Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time", where some self-attention heads and some MLP neurons are selected by running a simpler neural network.
- Lumiere: A Space-Time Diffusion Model for Video Generation (Paper Explained) by (4 February 2024) ► An opinionated presentation of "Lumiere: A Space-Time Diffusion Model for Video Generation".
- "MORE AGENTS" Is All You Need by (12 February 2024) ► "More Agents Is All You Need" analyses the gain of generating answers from several LLMs and using a voting mechanism to define the final answer.
- How Quickly Do Large Language Models Learn Unexpected Skills? — A new study suggests that so-called emergent abilities actually develop gradually and predictably, depending on how you measure them. by (13 February 2024) ► The debate about emergent capabilities appearing abruptly or continuously is still going on…
- Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch↑ by (18 February 2024) ► A clear description of LoRA and DoRA.
- How Selective Forgetting Can Help AI Learn Better — Erasing key information during training results in machine learning models that can learn new languages faster and more easily.↓ by (28 February 2024) ► Some limited and overly basic information about forgetting in language models, which are easier to train on new languages.
- A LoRA Successor, Small Finetuned LLMs Vs Generalist LLMs, and Transparent LLM Research by (3 March 2024) ► Can small fined-tuned models perform better on some tasks than large models, DoRA, OLMo is a real open-source model, Gemma…
- Tips for LLM Pretraining and Evaluating Reward Models — Discussing AI Research Papers in March 2024 by (31 March 2024) ► An analysis of continuous pretraining and a benchmark for evaluating reward models.
- How do mixture-of-experts layers affect transformer models? — This new LLM technique has started improving the results of models without additional training. by (4 April 2024) ► A short description of the Mixture of Experts architecture, I guess that if you know enough to understand this, you already know about MoE.
- Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping (Searchformer) by (6 April 2024) ► "Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping": using a transformer model to mimic an A* search algorithm, results are better when the model also has to reproduce the search traces.
- Why Recurrent Neural Networks are cursed | LM2 by (8 April 2024) ► A presentation of Recurrent Neural Networks.
- Flow Matching for Generative Modeling (Paper Explained) by (8 April 2024) ► "Flow Matching for Generative Modeling": a mathematical description of Flow Matching, a mechanism to train Continuous Normalising Flows.
- How did the Attention Mechanism start an AI frenzy? | LM3 by (15 April 2024) ► How the attention mechanism was implemented for RNN.
- Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention by (25 April 2024) ► A paper ("Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention") describing how to support infinite context length, but has some doubts about its real effectiveness.
- How AI 'Understands' Images (CLIP) - Computerphile by (25 April 2024) ► A basic presentation of CLIP.
- TransformerFAM: Feedback attention is working memory by (28 April 2024) ► Yet another paper ("TransformerFAM: Feedback attention is working memory") proposing infinite context by using additional tokens as a short term memory.
- ORPO: Monolithic Preference Optimization without Reference Model (Paper Explained) by (1 May 2024) ► A paper ("ORPO: Monolithic Preference Optimization without Reference Model") describing a method combining supervised fine-tuning and preference alignment.
- Shapley Values Explained | Interpretability for AI models, even LLMs! by (6 May 2024) ► A presentation of Shapley values, a method to explain how much each input impacts the model’s output, and an example with Llama 2 and the SHAP library.
- Has Generative AI Already Peaked? - Computerphile by (9 May 2024) ► A basic presentation of a paper ("No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance") claiming that multimodal models require exponentially more data to achieve linear improvements.
- GaLore EXPLAINED: Memory-Efficient LLM Training by Gradient Low-Rank Projection by (27 May 2024) ► Yet another pre-training/fine-tuning algorithm: "GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection".
- LLM Research Insights: Instruction Masking and New LoRA Finetuning Experiments — Discussing the Latest Model Releases and AI Research in May 2024 by (2 June 2024) ► Three papers: not masking instructions when calculating the loss for instruction finetuning performs better than masking, but only if the answer is short and the number of training examples is small; LoRA learns less and forgets less than full finetuning; MoRa is yet another finetuning algorithm.
- Machine Learning and Logistic Regression↓ by (19 July 2024) ► A bad description of logistic regression, the linear part is not explained.
- A New Type of Neural Network Is More Interpretable — Kolmogorov-Arnold Networks could point physicists to new hypotheses↓ by (5 August 2024) ► There is little valuable information about KAN networks in this article.
- Reinforcement Learning from Human Feedback (RLHF) Explained by (7 August 2024) ► The title says it all: a short presentation of RLHF.
- New LLM Pre-training and Post-training Paradigms — A Look at How Modern LLMs Are Trained by (17 August 2024) ► An overview and comparison of the pre- and post-trainings of Qwen 2, Apple Foundation Model, Gemma 2, and Llama 3.1.
- Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution – Paper Explained by (20 August 2024) ► "Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution" describes a diffusion model algorithm for text that gives correct results, before such models used to generate garbage text.
- What is Mixture of Experts? by (28 August 2024) ► A clear introduction to the MoE architecture and its advantages and challenges.
- Transformer LLMs are Turing Complete after all !? by and (5 September 2024) ► LLMs with chain-of-thought are equivalent to probabilistic Turing machines.
- Building A GPT-Style LLM Classifier From Scratch — Finetuning a GPT Model for Spam Classification by (21 September 2024) ► This article details how to fine-tune an LLM to use it as a classifier: replacing the output layer by a classifier one, then post-training it while freezing all layers except the last transformer block and the output layer.
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (Paper) by (5 October 2024) ► A review of "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters", a paper comparing the cost efficiency of best-of-n vs. beam search vs. lookahead search, and the cost efficiency of training vs. inference.
- Graph Language Models EXPLAINED in 5 Minutes! [Author explanation 🔴 at ACL 2024] by and (6 October 2024) ► proposes a method to build a Graph Language Model from an LLM.
- Text Classification: AI Techniques and Real-World Applications by (15 October 2024) ► A good but limited 101 presentation for managers: the basics of text classification.
- Understanding Multimodal LLMs — An introduction to the main techniques and latest models↑ by (3 November 2024) ► A good presentation of the architectures of models able to take both text and images as input. The two main options are Unified Embedding Decoder and Cross-modality Attention. describes ten recent such models.
- Large Language Models explained briefly by (20 November 2024) ► A short, basic, and clear presentation of how LLMs work.
- TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (Paper Explained) by (23 November 2024) ► A, rather negative, review of "TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters".
- Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty by (5 December 2024) ► A prototype of an image generation UI where the user is proposed to clarify some aspects of what s/he wants or s/he can edit an interpretable belief graph of the model.
- REPA Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You ... by (8 December 2024) ► Improving image generation diffusion models by combining them with vision transformer models.
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (Paper Explained) by (10 December 2024) ► Some comment on "Safety Alignment Should be Made More Than Just a Few Tokens Deep", a paper demonstrating that safety fine-tuning mostly impacts the first generated tokens, so you can get the model to generate unsafe content by controlling the first tokens, using a DAN prompt, fine-tuning it…
- Are LLMs capable of non-verbal reasoning? — Processing in the “latent space” could help AI with tricky logical questions. by (12 December 2024) ► The subtitle says it all.
- The Dark Matter of AI [Mechanistic Interpretability]↑ by (23 December 2024) ► A presentation of Mechanistic Interpretability and its use of Sparse Autoencoders.
- Byte Latent Transformer: Patches Scale Better Than Tokens (Paper Explained) by (24 December 2024) ► This description of "Byte Latent Transformer: Patches Scale Better Than Tokens" is difficult to understand.
- Noteworthy AI Research Papers of 2024 (Part One) — Six influential AI papers from January to June↑ by (31 December 2024) ► has chosen one interesting paper for each month of 2024. He gives some information extracted from each one.
- ↪Noteworthy AI Research Papers of 2024 (Part Two) — Six influential AI papers from July to December↑ by (15 January 2025) ► The second half of the year.
- Génération de données en IA par transport et débruitage (1) - Stéphane Mallat (2024-2025) by (15 January 2025) ► presents his lessons on AI data generation through transport and denoising.
- Training large language models to reason in a continuous latent space – COCONUT Paper explained by (26 January 2025) ► An overview of the Chain of Thoughts mechanism presented in "Training Large Language Models to Reason in a Continuous Latent Space".
- [GRPO Explained] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models by (26 January 2025) ► A description of TRPO, PPO, and GRPO. The most interesting is the conclusion that RL is only able to reshape the base model probabilities, not to add new capabilities to it.
- Diffusion Models for AI Image Generation by (30 January 2025) ► A clear, basic, and classic presentation of diffusion models.
- ChatGPT is made from 100 million of these [The Perceptron] by (1 February 2025) ► The history of gradient back-propagation.
- What is Semi-Supervised Learning? by (3 February 2025) ► The usual IBM’s presentation for manager, this time this is little information about semi-supervised learning.
- Speculative Decoding and Efficient LLM Inference with Chris Lott (⧉) by and (3 February 2025) ► The hardware constraints when running an LLM and some optimisations: KV caching, quantisation, pruning, speculative decoding…
- Understanding Reasoning LLMs — Methods and Strategies for Building and Refining Reasoning Models by (5 February 2025) ► A good overview of the current methods to build reasoning models: inference-time scaling, RL, SFT, and distillation.
- Elisa Fromont - Les modèles de diffusion by (6 February 2025) ► A mathematical description of the diffusion models.
- New AI text diffusion models break speed barriers by pulling words from noise — New diffusion models borrow technique from AI image synthesis for 10x speed boost. by (27 February 2025) ► Inception Labs has released Mercury Coder, an LLaDA (Large Language Diffusion with mAsking).
- How DeepSeek Rewrote the Transformer [MLA] by (5 March 2025) ► A presentation of DeepSeek’s Multi-Head Latent Attention.
- The State of LLM Reasoning Model Inference — Inference-Time Compute Scaling Methods to Improve Reasoning Models by (8 March 2025) ► summarises several articles among the current flurry of papers related to inference-time scaling.
- On the Biology of a Large Language Model (Part 1) by (5 April 2025) ► A presentation of two articles of Anthropic on interpretability: "Circuit Tracing: Revealing Computational Graphs in Language Models" and "On the Biology of a Large Language Model — We investigate the internal mechanisms used by Claude 3.5 Haiku — Anthropic's lightweight production model — in a variety of contexts, using our circuit tracing methodology.".
- ↪Exploring the "Biology" of LLMs with Circuit Tracing with Emmanuel Ameisen↑ (⧉) by and (14 April 2025) ► An interview with one of the authors of the previous papers.
- ↪On the Biology of a Large Language Model (Part 2) by (3 May 2025) ► is sarcastic on Anthopic’s snobbiness: they behave as if they are the only ones able to deal with AI complexity and impact. Nevertheless, they are doing good research.
- Introducing HELMET: Holistically Evaluating Long-context Language Models by , , , , , , and (16 April 2025) ► A presentation of the benchmark and the results of running it on 59 models.
- Direct Preference Optimization: A Technical Deep Dive by , , and (17 April 2025) ► A basic description of DPO.
- 4-Bit Training for Billion-Parameter LLMs? Yes, Really. by (18 April 2025) ► How to train a FP8 or FP4 model.
- The State of Reinforcement Learning for LLM Reasoning — Understanding GRPO and New Insights from Reasoning Model Papers by (19 April 2025) ► A description of RLHF, PPO, GRPO, and RLVR.
- Les 4 étapes pour entrainer un LLM by (25 April 2025) ► A basic description of the building of an LLM.
- ↪Les 4 étapes pour entrainer un LLM by (25 April 2025) ► Some additional information.
- The Strange Physics That Gave Birth to AI — Modern thinking machines owe their existence to insights from the physics of complex materials. by (30 April 2025) ► How had the idea to use models of spin glasses to build neural networks with memory.
- Language Concept Models: The Next Leap in Generative AI by (6 May 2025) ► After describing the usual encoder-decoder architecture, speaks about SONAR and Large Concept Models, but this is high-level introduction to the matter.
- The Misconception that Almost Stopped AI [How Models Learn Part 1] by (9 May 2025) ► This video tries to explain why gradient descent works, while we could believe it would be stuck in a local minimum, because there are a lot of dimensions.
- ↪The F=ma of Artificial Intelligence [Backpropagation, How Models Learn Part 2] by (11 June 2025) ► Yet another description of gradient backpropagation.
- ↪Why Deep Learning Works Unreasonably Well [How Models Learn Part 3]↑ by (10 August 2025) ► An explanation on why deeper neural networks give better results than wider ones.
- Token-Efficient Long Video Understanding for Multimodal LLMs | Paper explained by (18 May 2025) ► A presentation of STORM ("Token-Efficient Long Video Understanding for Multimodal LLMs"), a vision model using Mamba to compress video information.
- John Carmack (Keen Technologies): Research Directions @ Upper Bound 2025🚫 by and (20 June 2025) ► is getting bored, so he is trying to train neural networks to play Atari games in the real word using RL.
- Hot Swapping AI Skills: Attention Mechanisms & ALoRA Explained by (7 July 2025) ► A basic description of Activated LoRA (ALoRA).
- The Big LLM Architecture Comparison — From DeepSeek V3 to GLM-5: A Look At Modern LLM Architecture Design by (19 July 2025) ► Some technical details of some noticeable models released this year.
- ↪The Big LLM Architecture Comparison by (10 September 2025) ► The same as a video.
- But how do AI images and videos actually work? | Guest video by Welch Labs↑ by (25 July 2025) ► A good presentation of diffusion models.
- Greedy? Min-p? Beam Search? How LLMs Actually Pick Words – Decoding Strategies Explained by (3 August 2025) ► A summary of the decoding strategies: temperature, top-p, top-k, min-p, beam search…
- From GPT-2 to gpt-oss: Analyzing the Architectural Advances — And How They Stack Up Against Qwen3 by (9 August 2025) ► A list of optimisations done on the transformers architecture since GPT-2 (2019).
- These Numbers Can Make AI Dangerous [Subliminal Learning] by (4 September 2025) ► Using a toy model and simple maths to try to understand the subliminal effect (when a teacher model impacts a student model on subjects not appearing in the data generated by the first to teach the second).
- How Generative AI Video Works - Computerphile↓ by (5 September 2025) ► This presentation of video generation is rather unclear.
- Smol2Operator: Post-Training GUI Agents for Computer Use by , , , , and (23 September 2025) ► An open-sourced recipe to fine-tune a vision model into a GUI agent.
- SDS 929: Dragon Hatchling: The Missing Link Between Transformers and the Brain, with Adrian Kosowski (⧉) by and (7 October 2025) ► Yet another architecture aimed at supporting learning: Baby Dragon Hatchling (BDH). But this presentation is much too simplified, it is difficult to determine if this is real advancement or bullshit.
- Recurrence and Attention for Long-Context Transformers with Jacob Buckman (⧉) by and (7 October 2025) ► Yet another architecture trying to merge attention and recurrence: Power Retention.
- [Paper Analysis] On the Theoretical Limitations of Embedding-Based Retrieval (Warning: Rant) by (11 October 2025) ► complains that "On the Theoretical Limitations of Embedding-Based Retrieval", while being mathematically sound, speaks about an irrelevant hypothetical issue on the limitation of search using vector embeddings.
- nanochat by (13 October 2025) ► has implemented a new very small LLM that can be trained for $100.
- Cloning Yourself in AI using LoRA - Computerphile↓ by (16 October 2025) ► This presentation of LoRA is much too basic to be valuable.
- Flow-Matching vs Diffusion Models explained side by side by (19 October 2025) ► The title says it all.
- What the Books Get Wrong about AI [Double Descent] by (19 October 2025) ► The traditional idea of overfitting and the discovery of double descent.
- DeepSeek OCR - More than OCR by (20 October 2025) ► A presentation of the "DeepSeek-OCR: Contexts Optical Compression" paper which studies how to use fewer tokens by recording memory in image tokens rather than text tokens.
- LLM Building Blocks & Transformer Alternatives by (27 October 2025) ► A description of some optimisations to the transformer architecture and some alternatives to transformers.
- Google's new VaultGemma model – Differential Privacy explained by (2 November 2025) ► VaultGemma prevents samples appearing very few times from being memorised by clipping the gradient and adding noise to the average gradient of the batch.
- Beyond Standard LLMs — Linear Attention Hybrids, Text Diffusion, Code World Models, and Small Recursive Transformers by (4 November 2025) ► The description of some variations or alternatives to architecture based only on transformers: Linear Attention hybrids, Text Diffusion Models, Code World Models, and Small Recursive Transformers. The last ones are not about generating text.
- La Physique de l'Intelligence artificielle by (6 November 2025) ► The Ising model, Hopfield networks, and neural networks are converging towards behaving like a Gaussian process when they are getting larger, and their possible use to model quantum fields.
- Apriel-H1: The Surprising Key to Distilling Efficient Reasoning Models by , , , , and (19 November 2025) ► Some feedback on the training of a transformer reasoning model converted into a transformer/Mamba hybrid model.
- From DeepSeek V3 to V3.2: Architecture, Sparse Attention, and RL Updates — Understanding How DeepSeek's Flagship Open-Weight Models Evolved by (3 December 2025) ► The title says it all.
- Titans: Learning to Memorize at Test Time (Paper Analysis) by (14 December 2025) ► A presentation of "Titans: Learning to Memorize at Test Time", a paper presenting a model using a fully connected network as a memory store.
- TiDAR: Think in Diffusion, Talk in Autoregression (Paper Analysis) by (27 December 2025) ► A presentation of "TiDAR: Think in Diffusion, Talk in Autoregression", a paper presenting a variation of autoregressive models where the model predicts a few future tokens using a basic diffusion mechanism.
- What are State Space Models? Redefining AI & Machine Learning with Data by (15 January 2026) ► A high-level presentation of SSM, S4, Mamba; hybrid models…
- State of LLMs 2026: RLVR, GRPO, Inference Scaling — Sebastian Raschka by and (29 January 2026) ► An interesting summary of the AI models, in particular the transformer-based ones.
- Can humans make AI any better? by (31 January 2026) ► explains that LLMs are a dead end, AI’s training should not rely only on human knowledge.
- A Visual Guide to Attention Variants in Modern LLMs — From MHA and GQA to MLA, sparse attention, and hybrid architectures by (22 March 2026) ► An overview of the variations on the classic transformer architecture: Grouped-Query Attention, Multi-Head Latent Attention, Sliding Window Attention, DeepSeek Sparse Attention, Gated Attention, and Hybrid Attention.
- ↪A Visual Tour of Modern LLM Architectures by (28 March 2026) ► The same as a video. The video is not well-prepared and structured, but the work done by is impressive.
- The Race to Production-Grade Diffusion LLMs with Stefano Ermon (⧉) by and (26 March 2026) ► Some information about the diffusion language models from Inception Labs, such as Mercury 2, but there are few technical details.
- Recursion Is The Next Scaling Law In AI by and (1 May 2026) ► A technical presentation of Hierarchical Reasoning Models (HRM) and Tiny Recursive Models (TRM), but no pointers are provided to the two papers they talk about…
- What I Learned From Implementing LLM Architectures From Scratch (And How to Get Started) by (13 May 2026) ► explains how he re-implements models and describes some strategies to reduce the size of the KV cache.
- The data black hole at the center of AI — "We see these AIs as a galaxy glittering with capabilities, but at their center, invisible to the naked eye, holding all the constellations together, is an unimaginably massive black hole of data." (⧉) by (19 June 2026) ► Some thoughts about the fact that AI is very poor at learning, it needs an enormous data quantity to be trained on something.
- ↪The next big breakthrough will be AIs learning on the job — Labs are throwing away the most valuable data. (⧉) by (26 June 2026) ► RL still does not work for tasks. For things such as Computer Use, simulations may solve that. For other tasks, On-Policy Self-Distillation or Dreaming may be solutions to get LLMs to learn from the tasks they perform.
- Controlling Reasoning Effort in LLMs — How LLMs Learn Low-, Medium-, and High-Effort Reasoning Modes by (18 July 2026) ► A description of how models handle reasoning effort: they are trained to do so by SFT, RLVR, or hard token budgets.
- What Is AI Model Collapse? Why AI Could Forget Reality by (6 August 2026) ► A description of the model collapse problem (when model degrades generation after generation because they are trained on data they themselves generated), why it s a problem, and how to avoid it.
- OpenAI Astra and Recurrent Depth / Looped Transformers by (3 September 2026) ► A description of looped transformers / recurrent depth.
- Tokenisers
- Let's build the GPT Tokenizer⇈ by (20 February 2024) ► A very good description of tokenisation.
- So many tokens, so little time: Introducing a faster, more flexible byte-pair tokenizer — We released a new open source byte-pair tokenizer that is faster and more flexible than popular alternatives. by and (12 December 2024) ► The explanation of the algorithm is not so clear.
- Generative Adversarial Networks
- Generative Adversarial Networks (GANs) - Computerphile by (25 October 2017) ► Adversarial Networks and using them to generate images.
- Zebras, Horses & CycleGAN - Computerphile by (1 August 2019) ► A description of CycleGAN, two GANs working in opposite directions.
- Transformers
- AI Language Models & Transformers - Computerphile by (26 June 2019) ► The usage and implementation of language models, and the new attention-based ones: transformers.
- Transformers, explained: Understand the model behind GPT, BERT, and T5 by (19 August 2021) ► A basic presentation of transformers.
- Scaling Transformer to 1M tokens and beyond with RMT (Paper Explained) by (27 April 2023) ► An explanation of the "Scaling Transformer to 1M tokens and beyond with RMT" paper: a RNN where the base building block is a transformer.
- Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs by (14 January 2024) ► A clear and detailed description of self-attention implementation.
- Transformers explained | The architecture behind LLMs by (21 January 2024) ► Yet another explanation of the transformer architecture. This one is correct.
- SDS 759: Full Encoder-Decoder Transformers Fully Explained, with Kirill Eremenko (⧉) by and (20 February 2024) ► Yet another presentation of transformers, this one is so-so.
- À quoi ressemble ChatGPT ? 🌶️↓ by (29 October 2024) ► This explanation of transformers is far too fast to be understandable.
- [Paper Analysis] The Free Transformer (and some Variational Autoencoder stuff) by (1 November 2025) ► A presentation of "The Free Transformer", a variation of the transformer architecture that embeds random latent variables in order to condition the token generation.
- MAMBA
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Paper Explained) by (24 December 2023) ► The paper presenting the Mamba architecture: "Mamba: Linear-Time Sequence Modeling with Selective State Spaces".
- This CRAZY Paper on Mamba has got some REAL Juice!!! by (7 February 2024) ► A shallow presentation of the "Repeat After Me: Transformers are Better than State Space Models at Copying" paper.
- MAMBA and State Space Models explained | SSM explained by (17 February 2024) ► Another presentation of MAMBA.
- The FIRST Production-grade Mamba-based LLM!!! by (31 March 2024) ► A presentation of ai21labs/Jamba-v0.1, a hybrid Mamba/Transformer architecture.
- Attention!!! JAMBA Instruct - Mamba LLM's new Baby!!! by (3 May 2024) ► Jamba-Instruct, a chatbot based on Jamba.
- xLSTM
- xLSTM: Extended Long Short-Term Memory by (2 June 2024) ► A presentation of "xLSTM: Extended Long Short-Term Memory", a study of LSTM variants scaled to billions of parameters.
- xLSTM Explained in Detail!!! by and (1 July 2024) ► , an author of the previous paper, is presenting xLSTM.
- Ternary models
- Floating Points are no more, Changes everything for LLMs!!!↓ by (28 February 2024) ► "The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits" describes an LLM using only -1, 0, and 1 as weights.
- Scalable MatMul-free Language Modeling (Paper Explained) by (8 July 2024) ► presents and comments "Scalable MatMul-free Language Modeling", a paper proposing to replace matrix multiplication with ternary weights.
- Understanding 1.58-bit Large Language Models🚫 by (7 September 2024) ► A good presentation of the state of the art of ternary models.
- Microsoft’s “1‑bit” AI model runs on a CPU only, while matching larger systems — Future AI might not need supercomputers thanks to models like BitNet b1.58 2B4T. by (18 April 2025) ► Some basic information about Microsoft’s BitNet b1.58 2B4T.
- Energy-Based Transformers
- Energy-Based Transformers are Scalable Learners and Thinkers (Paper Review) by (19 July 2025) ► A presentation of Energy-Based Transformers ("Energy-Based Transformers are Scalable Learners and Thinkers").
- Energy-Based Transformers explained | How EBTs and EBMs work by (21 September 2025) ► The same.
- PyTorch
- Débuter avec PyTorch by (18 March 2021) ► A short introduction to Pytorch with a small example.
- Building a Single Layer Neural Network in PyTorch by (8 April 2023) ► A complete and very simple example.
- SDS 819: PyTorch: From Zero to Hero, with Luka Anicin (⧉) by and (17 September 2024) ► provides a basic presentation of Pytorch and describes how he got into machine learning.
- Making GPUs Actually Fast: A Deep Dive into Training Performance by and (11 November 2025) ► How to avoid GPU starvation: avoiding CPU/GPU synchronisations, kernel fusion, writing custom kernels in CUDA…
- Get SH*T Done with PyTorch
- Getting Started with PyTorch by (6 February 2020) ► A short introduction to PyTorch.
- Build Your First Neural Network with PyTorch by (21 February 2020) ► A simple neural network.
- Transformers
- The Transformers Library: standardizing model definitions by , , , and (15 May 2025) ► Hugging Face wants its library to be usable across the whole AI landscape.
- Transformers v5: Simple model definitions powering the AI ecosystem by , , , and (1 December 2025) ► Transformers 5.0.0 has a first release candidate. This article is an overview of the current status of the library.
- 3blue1brown’s Deep learning
- But what is a neural network? | Deep learning chapter 1 by (5 October 2017) ► A basic introduction to the structure of neural networks and how such a structure could work.
- Gradient descent, how neural networks learn | Deep Learning Chapter 2 by (16 October 2017) ► Training a neural network consists of minimising a cost function and how to use gradient descent to perform this minimisation.
- Backpropagation, intuitively | Deep Learning Chapter 3 by (3 November 2017) ► Getting a feeling of how backpropagation works.
- Backpropagation calculus | Deep Learning Chapter 4 by (3 November 2017) ► The calculus expressions of backpropagation.
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 by (1 April 2024) ► A very high-level description of an LLM architecture and the description of embedding and softmax.
- Attention in transformers, step-by-step | Deep Learning Chapter 6 by (7 April 2024) ► A description of the transformer architecture.
- Visualizing transformers and attention | Talk for TNG Big Tech Day '24 by (5 July 2024) ► A 45-minutes summary of the previous videos.
- How might LLMs store facts | Deep Learning Chapter 7↑ by (31 August 2024) ► A description of how the multilayer perceptrons complete embedded vectors with related data during inference.
- Grokking
- Grokking: Generalization beyond Overfitting on small algorithmic datasets (Paper Explained) by (6 October 2021) ► A paper ("Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets") on the grokking phenomenon where generalisation seems to happen abruptly and long after fitting the training data.
- "Grokking" : les modèles d'IA sont-ils capables de piger ? — Ce phénomène étonnant, découvert récemment, pourrait changer notre compréhension de l'apprentissage et de la cognition dans les réseaux de neurones... by (11 September 2023) ► Another presentation of grokking.
- How Do Machines ‘Grok’ Data? — By apparently overtraining them, researchers have seen neural networks discover novel solutions to problems. by (12 April 2024) ► Some researchers have understood some cases of grokking, but the phenomenon has only been studied from small neural networks doing modular arithmetic.
- The most complex model we actually understand↑ by (20 December 2025) ► A detailed explanation of the well-known grokking effect happening when a model is trained to perform modular addition.