In a large bowl, whisk condensed milk, whole milk, and pudding mix, breaking up any lumps. Refrigerate until set, about 5 minutes.

Meanwhile, in another large bowl, using a handheld mixer on medium-high speed, beat cream and vanilla until stiff peaks form, 2 to 3 minutes. Set aside half of whipped cream for serving. Fold remaining whipped cream into milk mixture.

Cover bottom of 3-quart trifle dish with wafers. Top with one-third of pudding mixture. Cover with another layer of wafers. Top with an even layer of banana slices. Continue to layer pudding, wafers, and bananas until you reach the top, ending with a final layer of pudding. Set aside remaining wafers for serving.

Refrigerate at least 3 hours or up to overnight.

Add granulated sugar to reserved whipped cream and stir to combine. Dollop on top of banana pudding. Crumble reserved wafers, then sprinkle over pudding.