188: World Models

**Intro topic: Running** **News/Links:** - Flow matching versus diffusion models - https://youtu.be/firXjwZ\_6KI?is=QMq8DcCsXTTktOuE - OpenCV 5 - https://opencv.org/opencv-5/ - Claude fable beats pokemon with no harness - https://youtu.be/Ty\_50J84fMY?si=EJ1KjZCZegipfCsV - Can the stockmarket swallow Anthropic, SpaceX and OpenAI?  - https://archive.ph/nKEVw **Book of the Show** - Patrick - Strength of the Few - James Islington - https://amzn.to/4pmr10T - Jason - Descender - Jeff Lemire - https://amzn.to/3QKhp3l Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h **Tool of the Show** - Patrick - No Man’s Sky - Jason - Paperlib https://paperlib.app/en/ **Topic: World Models** - Making decisions with AI - Action-Value (called a Q model): What is the long-term value of making a decision at a position - Policy: What action should I take (must be a distribution) - Value (called a V model): What is the value of a position (depends on policy) - Advantage/Disadvantage: difference in value given two policies - When advantage is +, do that more. - Model-Free - Look at the current situation and suggest an action - Run that action in the real world and measure the effect - Use that measurement to suggest better actions next time - Model-Based - Observe rollouts (sequences of situations) and learn the dynamics - Choose an action, use your dynamics model to measure the consequence - Potentially do MPC (try many actions and choose the best) - World Models - Observe many many rollouts and learn a full forward model (how to create the input in the future) - Train a policy & value inside the world model - Deploy the policy and fine-tune based on the real world