Place a large bowl of ice and water next to stovetop. Bring 4 cups water to a boil in a medium pot over high; reduce heat to medium-low. Squeeze lemon juice into water, and add lemon halves to pot. Stir in salt and 1 teaspoon Old Bay.

Add shrimp to pot; reduce heat to low, and simmer gently until shrimp are bright pink and just cooked through, 2 to 3 minutes. Transfer shrimp using a slotted spoon to prepared ice bath, and let cool; drain. Roughly chop shrimp into bite-size pieces.

Stir together celery and celery leaves, bell pepper, chives, tarragon, and mint in a large bowl until combined. Stir shrimp and remaining 1 teaspoon Old Bay into vegetable mixture; toss to coat.

Stir in mayonnaise and Dijon mustard, stirring mixture until evenly coated and well combined. Cover bowl with plastic wrap, and refrigerate for about 1 hour before serving. Season with additional salt to taste, and garnish with additional celery leaves.