Research Paper Deep Dive - The Sparsely-Gated Mixture-of-Experts (MoE)

In this video we are taking a deep dive to learn the more about the Mixture of Experts (or MoE), how it works and internal architecture, text and images data processing..

More Info:
-----------------
We all know that large model sizes is necessary for strong generalization and robustness, so training large models while limiting resource requirements is becoming increasingly important.

There is a hidden problem underneath these superstar, eye-popping results and the problem is significant use of computation resources or the requirements, which includes supermassive hardware and that includes logistics, cost, power requirement, and top of the above feasibility to even move it outside labs..

GitHub Resources:
https://github.com/prodramp/DeepWorks/tree/main/MoE

Research Paper and Code:
https://arxiv.org/abs/1701.06538
https://github.com/davidmrau/mixture-of-experts

▬▬▬▬▬▬ ⏰ TUTORIAL TIME STAMPS ⏰ ▬▬▬▬▬▬
- (00:00) Paper Introduction
- (00:22) Understanding the Problem
- (01:22) Significant computation requirement
- (02:33) Solution - Conditional Computation
- (03:35) Dense and Sparse Models
- (04:17) Pathway Models
- (06:20) MoE Introduction
- (08:38) MoE Internals
- (11:05) MoE Components
- (14:44) Data Processing in MoE
- (15:09) Text Data Processing in MoE
- (17:07) Image Data Processing in MoE
- (19:47) Text and Image Data Processing in MoE
- (20:46) Research Paper and Code
- (21:30) Resources and GitHub Reference

Connect
------------------
- Prodramp LLC (@prodramp)
- Website - https://prodramp.com
- LinkedIn - https://www.linkedin.com/company/prodramp
- GitHub- https://github.com/prodramp/
- AngelList - https://angel.co/company/prodramp
- Facebook - https://www.facebook.com/Prodramp

Content Creator: Avkash Chauhan (@avkashchauhan)
- https://www.linkedin.com/in/avkashchauhan
- https://twitter.com/avkashchauhan

Tags:
#moe #ai #cnn #ml #lime #aicloud #h2oai #driverlessai #machinelearning #cloud #mlops #model #collaboration #deeplearning #modelserving #modeldeployment #pytorch #datarobot #datahub #streamlit #modeltesting #codeartifact #dataartifact #modelartifact #onnx #aws #kaggle #mapbox #lightgbm #xgboost #classification #dataengineering #pandas #keras #tensorflow #tensorboard #cnn #prodramp #avkashchauhan #LIME #mli #xai

PandasNumber SeriesOpenAI