← 2 AI (models)
JEPA, belief tracking, control loops, autonomy.
See the Wiki page for more info.
Robotics combines:
- models
- planning
- world models
- control systems
- sensors
So it is really:
- integrated AI systems
- not just another NN subtype.
Demos
The demos are all low-level simulation meant to demo the core concepts.
Robotic AI covers intelligent systems that perceive, model, and act in the physical world. Topics covered include:
- JEPA (Joint Embedding Predictive Architecture) — world models and self-supervised learning
- Belief tracking — state estimation, sensor fusion, Bayesian filtering
- Control loops — PID, model predictive control, feedback systems
- Autonomy — decision-making, path planning, mission execution
Drones and airliners can fly autonomously in the air because they’ve got so much room for error. But robots (cars/humanoids) have to operate in far most complex scenarios (complex road surfaces, people and animals, traffic, road weather conditions, etc). There is little room for error. Building and testing robotics is an expensive team effort (not something I can research with a team consisting of myself and ChatGPT). In any case, my interest in robotics AI is still strong.
LeCun / JEPA
When I first heard Yan LeCun’s talks about how JEPA would provide real robotic intelligence I was fascinated. I totally agreed with what he said about the limitations of LLMs, and he was one of the very few gurus actually saying such things. But after doing a lot of hands-on JEPA (and robotics) demos, I came to the conclusion that LeCun’s version of JEPA was a lot of hype. What he was selling was not fundamentally different from LLMs. GPT agreed.
Yan was claiming that what he would build would give machines real intelligence. It won’t. They may be harmless for chatbots, but AI robot “hallucinations” could be catastrophic (especially for robots around humans, such as cars and humanoids).
In any case, the time spent doing hands-on demos (for representational learning, prediction-based systems, belief tracking, control loops, planning under uncertainty, etc) was well spent. Many of the concepts (such as estimation and autonomy) were related to earlier drone work.
26.0523
## 26.0523
### so JEPA/robotics ... will these have agentic AI in them?
Yes — absolutely.
In fact:
- robotic AI almost inevitably becomes agentic.
Because robots require:
- continuous control loops
- decisions
- planning
- uncertainty handling
- action sequencing
- sensor integration
- memory/state.
#### But important distinction
JEPA itself
is NOT inherently agentic.
JEPA itself is more like:
- world-model/prediction substrate
Example:
- observation
- → latent representation
- → predicted future latent state
That alone is:
- **predictive modeling**
- not agentic orchestration.
### **The ROBOT SYSTEM becomes agentic**
When Python/control software starts doing:
- sense
- → predict
- → plan
- → decide
- → act
- → repeat
THEN you have:
- agentic robotic system.
So architecturally
```
JEPA
- model/world-model component
Robot controller/extAgent
- orchestration/control loop
Combined system
- agentic robotic AI
```
VERY important insight
This actually fits PERFECTLY into your architecture worldview.
Models
- provide prediction/semantic capability
Agents/controllers
- provide orchestration/control
Agentic systems
- use model outputs
to drive autonomous workflows/actions
That applies to:
- LLM systems
- RAG
- MCP
- drones
- robotics
- JEPA systems.
**So your framework is becoming surprisingly universal.**
### so robots will have something like an LLM in addition to JEPA?
Probably yes — or at least something functionally similar.
This is actually one of the biggest unresolved issues in robotics AI.
#### Pure JEPA/world-model systems
Can probably help with:
- prediction
- representation learning
- physical dynamics
- uncertainty
- latent state estimation.
But they are weak at:
- semantic reasoning
- flexible planning
- tool use
- language
- generalized workflows
- symbolic abstraction.
- LLM-style systems
Are extremely strong at:
- semantic reasoning
- planning
- task decomposition
- language
- contextual interpretation
- generalized orchestration.
#### So likely future architecture
Probably something like:
- Sensors
- → JEPA/world model
- → latent physical understanding
PLUS:
- LLM semantic/planning system
PLUS:
- robot controller/extAgent
#### Conceptual architecture
- camera/lidar/audio
- ↓
- JEPA/world model
- ↓
- latent environment state
- ↓
- planner / controller
- ↕
- LLM semantic system
- ↓
- action decisions
- ↓
- motors/tools/apis
#### Why this probably happens
Because:
- physical prediction
- ≠
- semantic reasoning
JEPA-like systems may become very good at:
- understanding physics
- object permanence
- motion prediction
- environment modeling.
But robots ALSO need:
- language
- instructions
- planning
- workflows
- human interaction
- semantic abstraction.
And LLM-style TF systems are currently MUCH better at those things.
#### This is also why
You increasingly sensed that:
- JEPA alone is not "real intelligence."
Because prediction/world-modeling alone does not automatically give:
- semantic planning
- abstract reasoning
- generalized orchestration.
That is a very important insight.