Make the fritters: combine the white cheddar, yellow cheddar, cream cheese, mayonnaise, and pimento peppers. Chill in fridge for at least 2 hours.
Form the mixture into 1-inch balls (a melon scooper is helpful). Place onto parchment lined baking sheet.
Place flour, buttermilk, and panko into 3 separate dishes.
Roll each ball in the flour, shaking off any excess flour. Then dip in buttermilk. Then roll around in panko. Repeat buttermilk then panko to make sure the whole ball is covered. If not it will leak out while frying. Repeat until all balls are covered.
Return to fridge for 30 minutes.
Heat oil in fryer or in large pot to 350 degrees. Fry the pimento cheese balls 5-7 at a time - not overcrowding the fryer/pan. Fry for 3-4 minutes, until golden brown. Remove with skimmer and place on paper towel lined plate. Repeat until all are fried.
Serve with pepper jelly and crackers.