The Blues took a seventh-minute lead at Chigwell Construction Stadium when Billy Gilmour scored from the penalty spot after Reece Oxford had upended Brown in the area.
Within five minutes the hosts were level from a spot-kick off their own. Joe Powell converted the opportunity after Grady Diangana had gone down.
The Hammers were reduced to 10 men early in the second half when Tunji Akinola received a second booking.
And Chelsea took full advantage with Brown producing a composed finish into the bottom corner midway through the second half.
The Blues, whose senior side visit West ham in the Premier League on Sunday, move level on 11 points with leaders Arsenal after the opening six league matches of the campaign.