Adjust an oven rack to the middle position and heat the oven to 200°F. Set a wire rack inside a rimmed baking sheet—this prevents the finished sandwiches from getting soggy while they wait. In a small bowl, whisk together the mayonnaise, garlic, and rosemary. In a separate large bowl, toss the mushrooms with 4 tablespoons of the oil, salt, and pepper until well coated.

Heat a 12-inch nonstick grill pan over medium heat for about 1 minute. Place the mushrooms in the pan, gill-side up, and set a heavy Dutch oven directly on top of them. Cook until the mushrooms are well browned and tender, about 5 minutes per side. Transfer the mushrooms to a plate and wipe out the pan with paper towels.

Brush the remaining 2 tablespoons of oil evenly over one side of each slice of bread. Flip the slices over and spread the mayonnaise mixture over the second side. Assemble the sandwiches by layering half of the fontina, the cooked mushrooms, red peppers, and the remaining fontina between the bread, ensuring the mayonnaise side faces the filling. Press down gently to set the layers.

Return the grill pan to medium heat for 1 minute. Place two sandwiches in the pan, top with the Dutch oven to press, and cook until the bread is golden and crisp on the first side, about 4 minutes. Flip the sandwiches, replace the Dutch oven, and cook until the second side is crisp and the cheese has fully melted, about 4 minutes.

Transfer the finished sandwiches to the wire rack in the warm oven. Wipe out the grill pan and repeat the process with the remaining two sandwiches. Serve hot while the cheese is still pulling.