Sadio Mane opened the scoring on 40 minutes, slotting in from 12 yards after being played in by Milner to score his 50th goal in 100 Premier League appearances for Liverpool.
Mane was unable to double the Premier League leaders' advantage just before half-time, shooting straight at Kasper Schmeichel.
Leicester equalised on 80 minutes when substitute Ayoze Perez set up James Maddison, who fired low under Adrian.
Milner struck the decisive penalty in the fifth minute of stoppage time after Mane was fouled by Marc Albrighton.
Liverpool are on 24 points, eight above second-placed Manchester City having played one match more, after winning each of their first eight fixtures of the league campaign.
Brendan Rodgers' Leicester are 10 points behind in third.
Liverpool: 20 Oct v Man Utd (A)
Leicester: 19 Oct v Burnley (H)