REFINE-DP
Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning

Abstract

Humanoid loco-manipulation requires coordinated high-level motion plans with stable, low-level whole-body execution under complex robot-environment dynamics and long-horizon tasks. While diffusion policies (DPs) show promise for learning from demonstrations, deploying them on humanoids poses critical challenges: the motion planner trained offline is decoupled from the low-level controller, leading to poor command tracking, compounding distribution shift, and task failures. The common approach of scaling demonstration data is prohibitively expensive for high-dimensional humanoid systems. To address this challenge, we present REFINE-DP (REinforcement learning FINE-tuning of Diffusion Policy), a hierarchical framework that jointly optimizes a DP high-level planner and an RL-based low-level loco-manipulation controller. The DP is fine-tuned via a PPO-based diffusion policy gradient to improve task success, while the controller is simultaneously updated to accurately track the planner's evolving command distribution, reducing the distributional mismatch that degrades motion quality. We validate REFINE-DP on a humanoid robot performing long-horizon loco-manipulation tasks, including door traversal and long-horizon object transport. REFINE-DP achieves over $90\%$ success rate in simulation, even in extreme out-of-distribution cases (with respect to the pre-trained policy), and enables smooth autonomous task execution in real-world dynamic environments. Our proposed method substantially outperforms pre-trained DP baselines and demonstrates RL fine-tuning is key to reliable humanoid loco-manipulation.

Method

REFINE-DP Framework: The pipeline consists of three stages. (a) Data Collection: expert demonstrations are gathered via human teleoperation and heuristic behavior planners, executed through a frozen low-level RL loco-manipulation controller. (b) Pre-Training the High-level Planner: a diffusion policy with a transformer backbone is trained on the collected offline dataset to generate base velocity and hand pose commands. (c) REFINE-DP Fine-tuning: the pre-trained diffusion policy and the low-level RL controller are jointly optimized via PPO in simulation, maintaining distributional consistency between the planner and controller and substantially improving task success rates.

Three-Stage Pipeline

1

Stage 1: Data Collection

2

Stage 2: Diffusion Policy Pre-training

3

Stage 3: REFINE-DP Joint Fine-tuning

Low-level RL Loco-manipulation Controller

Hardware Demonstrations

We validate REFINE-DP on a Booster T1 humanoid robot (29 DoF) across four challenging loco-manipulation tasks requiring coordinated whole-body control.

Task 1: Walk and Pick Up a Box

×2

Trial 1

×2

Trial 2

Task 2: Walk, Pick Up an Object, and Walk to Place It

×2

Trial 1

×2

Trial 2

Task 3: Walk and Open a Door

×2

Trial 1

×2

Trial 2

Task 4: Step Up and Retrieve an Object

×2

Trial 1

×2

Trial 2

Quantitative Results

Each method is evaluated over 100 trials per task in simulation. REFINE-DP (FT DiT Planner & Joint Optimization) consistently outperforms all baselines across every task.

Joint Optimization Improves Fine-tuning Efficiency and Motion Quality

Optimizing the low-level controller yields two benefits: it alone boosts success rate by 18% on the long-horizon pick-and-place task, and jointly optimizing it halves the fine-tuning iterations needed (20 vs. 40) to reach 90% success.

Tracking Performance and End-Effector Velocity Across Tasks (upper-body): Joint optimization achieves the lowest position error and cuts orientation error by up to 50% vs. the pre-trained policy, while reducing end-effector velocity by 15% on average, allowing the joint optimized WBC policy $\pi'_{\text{loco-manip}}$ to achieve smoother manipulation. Fine-tuning the diffusion policy alone can degrade tracking, showing that adapting the planner in isolation is insufficient; the policy instead adapts to poor tracking accuracy by over-commanding.

Data Efficiency: Scaling vs. Fine-tuning

Data Efficiency: Training from scratch requires ~1000 trajectories to reach 90% success. A pre-trained diffusion policy fine-tuned with REFINE-DP achieves 95–97% success from just 50 demonstrations, a 20× reduction in data requirements.

Fine-tuning Over Pre-Trained Policies

Fail and Successful Trials

×2

Fail Door Opening

×2

Sucessful Door Opening

×2

Fail Box Transportation

×2

Sucessful Box Transportation

Increase in Task Throughput

Fine-tuned policies learn action efficiency and eliminate the indecisive motions common in pre-trained policies, reducing average task completion time by up to 26% across tasks in simulation. In hardware experiments, fine-tuned policy achieve an average time reduction of around 10% and 20% for the box-transporting and door-opening task respectively.

×2

Pre-trained Box-Pickup

×2

Fine-tuned Box-Pickup

×2

Pre-trained Door-Opening

×2

Fine-tuned Door-Opening

Appendix

Observation Terms for RL Policy Training