Palmer rejoins the club he was on loan with last season after agreeing a new four-year contract at Stamford Bridge.
The 20-year-old is Huddersfield's fifth signing of the summer. He follows Laurent Depoitre, Aaron Mooy, Jonas Lossl and Tom Ince.
"I’m very happy to have Kasey back at the club," Huddersfield head coach David Wagner told htafc.com.
“He has real desire and hunger to be as good as he can be, but he’s also very humble. This is the best and most exciting mix that a player can have as he strives to become an excellent Premier League player."
England Under-21 international Palmer recorded five goals and three assists last season to help Huddersfield to promotion to the Premier League for the first time.
Palmer could make his PL debut at Crystal Palace on the opening weekend.