The Argentinian's final match in charge was the 3-0 defeat at Newcastle United on Saturday, a result that left the club fourth from bottom and a point above the relegation zone.
Pellegrino, 46, moved to the south coast last summer but, after a promising start, Southampton had claimed just one win in their last 17 Premier League matches.
Assistant manager Carlos Compagnucci and assistant first-team coach Xavier Tamarit have also left the club.
Southampton say they will look to appoint a new management team "as soon as possible", with the search for a replacement already under way.
The team are next in Premier League action at West Ham United on 31 March.