Brown will spend 2017/18 with the Seagulls after agreeing a new four-year contract at Stamford Bridge.
Brighton manager Chris Hughton said: "He is a player with great potential and will bring a different dimension to our attacking options for the season ahead, alongside our existing strikers."
The 20-year-old is Brighton’s third senior summer signing. He follows Pascal Gross from Ingolstadt and Mathew Ryan from Valencia.
Last season he helped Huddersfield Town gain promotion to the Premier League.
A former Premier League 2 winner in 2013/14, Brown also became the second-youngest debutant in Premier League history when he played for West Bromwich Albion against Wigan Athletic aged 16 years and 117 days in May 2013.
Brown could make his Brighton PL debut at home to Manchester City on the opening weekend.