Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling.

Wed, 01 Jan 2025 00:00:00 +0100

Junfeng Ran, Guangxiang Zhao, Yuhan Wu, Dawei Zhu, Longyun Wu, Yikai Zhao, Tong Yang, Lin Sun, Xiangzheng Zhang, Sujian Li:
Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling. ECAI 2025: 4241-4248