This is my participation 11 The fourth of the yuegengwen challenge 3 God .

Use OpenCV and Python Identify credit card number

In the previous post , We learned how to install Tesseract Binary file and use it for OCR. Then we learned how to use basic image processing techniques to clean up images to improve Tesseract OCR Output .

however , Should not Tesseract It is regarded as a general-purpose device that can obtain high-precision optical character recognition 、 Off the shelf solutions . In some cases , It will work well —— And in other cases , It will fail miserably . A good example of this use case is credit card recognition , Given input image , We hope ：

Localize four groups of four digits , It's related to the sixteen digits on the credit card . application OCR Identify the sixteen digits on the credit card . Identify the type of credit card （ namely Visa、MasterCard、American Express etc. ）.

In these cases ,Tesseract Library does not recognize numbers correctly （ This may be because Tesseract No training in credit card sample Fonts ）. therefore , We need to OCR Credit card design our own custom solution . In today's blog post , I'll show you how to use template matching as OCR To help us create a solution to automatically identify credit cards and extract relevant credit card numbers from images .

Today's blog post is divided into three parts . In the first part , We will discuss OCR-A typeface , This is a font specially created to assist optical character recognition algorithm . Then we will design a computer vision and image processing algorithm , It can ：

• Localize the four groups of four digits on the credit card .
• Extract each of the four groups , Then separate 16 Each of the numbers .
• Use templates to match and OCR-A Font recognition 16 Each of the credit card numbers .

Last , We'll look at some credit cards OCR Examples of the algorithm applied to real images .

Through and with OpenCV Template matching OCR

In this section , We will use Python + OpenCV Implement our template matching algorithm to automatically identify credit card numbers .

To achieve this , We need to apply many image processing operations , Including thresholds 、 Calculate the gradient amplitude representation 、 Morphological operation and contour extraction . These techniques have been used in other blog posts to detect barcodes in images and identify machine-readable areas in passport images .

Since many image processing operations will be applied to help us detect and extract credit card numbers , So I included a lot of intermediate screenshots when the input image passed through our image processing pipeline .

These additional screenshots will give you a deeper understanding of how we can link basic image processing technologies together to build solutions for computer vision projects . Let's get started .

Open a new file , Name it ocr_template_match.py , We'll start working ：

# import the necessary packages
from imutils import contours
import numpy as np
import argparse
import imutils
import cv2
 Copy code 

To install / upgrade imutils , Just use pip ：

pip install --upgrade imutils
 Copy code 

Be careful ： If you use Python A virtual environment （ Like all my OpenCV Like the installation tutorial ）, Make sure to use... First workon Command to access your virtual environment , Then install / upgrade imutils .

Now we have installed and imported the package , We can parse our command line arguments ：

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True,
	help="path to input image")
ap.add_argument("-r", "--reference", required=True,
	help="path to reference OCR-A image")
args = vars(ap.parse_args())
 Copy code 

A parameter parser is established , Add two parameters , Then parse them , Store them as variables args . The two required command line arguments are ：

--image ： To carry out OCR The path of the processed image .

--reference ： Reference resources OCR-A Image path . The image contains OCR-A The number in the font 0-9, This allows us to perform template matching later in the pipeline .

Next, let's define the type of credit card ：

# define a dictionary that maps the first digit of a credit card
# number to the credit card type
FIRST_NUMBER = {
	"3": "American Express",
	"4": "Visa",
	"5": "MasterCard",
	"6": "Discover Card"
}
 Copy code 

Credit card type , For example, American Express 、Visa etc. , It can pass the inspection 16 The first digit in the credit card number to identify . We defined a dictionary FIRST_NUMBER , It maps the first number to the corresponding credit card type . Let's load the reference OCR-A Image to start our image processing pipeline ：

# load the reference OCR-A image from disk, convert it to grayscale,
# and threshold it, such that the digits appear as *white* on a
# *black* background
# and invert it, such that the digits appear as *white* on a *black*
ref = cv2.imread(args["reference"])
ref = cv2.cvtColor(ref, cv2.COLOR_BGR2GRAY)
ref = cv2.threshold(ref, 10, 255, cv2.THRESH_BINARY_INV)[1]
 Copy code 

First , We load reference OCR-A Images , It is then converted to grayscale and threshold + reverse . In each of these operations , We store or overwrite ref , Our reference image .

Now let's be in OCR-A Locate the outline on the font image ：

# find contours in the OCR-A image (i.e,. the outlines of the digits)
# sort them from left to right, and initialize a dictionary to map
# digit name to the ROI
refCnts = cv2.findContours(ref.copy(), cv2.RETR_EXTERNAL,
	cv2.CHAIN_APPROX_SIMPLE)
refCnts = imutils.grab_contours(refCnts)
refCnts = contours.sort_contours(refCnts, method="left-to-right")[0]
digits = {}
 Copy code 

Found the outline in the reference image . then , because OpenCV 2.4、3 and 4 How versions store the returned contour information differently , We check the version and correct refCnts Make appropriate changes . Next , We sort the contours from left to right , And initialize a dictionary ,digits, It maps numeric names to regions of interest .

here , We should traverse the contour , extract ROI And associate it with its corresponding number ：

# loop over the OCR-A reference contours
for (i, c) in enumerate(refCnts):
	# compute the bounding box for the digit, extract it, and resize
	# it to a fixed size
	(x, y, w, h) = cv2.boundingRect(c)
	roi = ref[y:y + h, x:x + w]
	roi = cv2.resize(roi, (57, 88))
	# update the digits dictionary, mapping the digit name to the ROI
	digits[i] = roi
 Copy code 

Traverse the reference image contour .

In circulation , i Save numeric name / Number , c Save outline . We surround each contour c Calculate a bounding box , For storing rectangles (x, y) Coordinates and width / Height . Use the bounding rectangle parameter from ref（ Reference image ） Extract from roi. The ROI Contains figures .

We will ROI Size to 57×88 Fixed size of pixels . We need to make sure that each number is adjusted to a fixed size , To apply template matching in number recognition later in this tutorial .

We will each number 0-9（ Dictionary key ） With each roi Images （ The dictionary is worth ） Related to .

At this point , We have completed the work of extracting numbers from the reference image and associating them with the corresponding number names .

Our next goal is to isolate the input --image Medium 16 A credit card number . We need to find and isolate the numbers first , Then you can start template matching to identify each number . These image processing steps are very interesting and insightful , Especially if you've never developed an image processing pipeline before , Please pay close attention to .

Let's continue to initialize a few structured kernels ：

# initialize a rectangular (wider than it is tall) and square
# structuring kernel
rectKernel = cv2.getStructuringElement(cv2.MORPH_RECT, (9, 3))
sqKernel = cv2.getStructuringElement(cv2.MORPH_RECT, (5, 5))
 Copy code 

You can think of the kernel as a small matrix that we slide over the image , To carry out （ Convolution ） operation , For example, blur 、 sharpening 、 Edge detection or other image processing operations .

Two such cores are constructed —— A rectangle and a square . We will use a rectangle as Top-hat Morphological operator , Use square as closed operation . We'll see this soon . Now let's get ready for OCR Image ：

# load the input image, resize it, and convert it to grayscale
image = cv2.imread(args["image"])
image = imutils.resize(image, width=300)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
 Copy code 

Loaded the command line parameter image containing the credit card photo . then , We adjust it to width=300 , Keep the aspect ratio , Then convert it to grayscale . Let's look at our input image ：

Next is our resizing and grayscale operation ：

Now our image has been grayed and the size is the same , Let's do morphological operations ：

# apply a tophat (whitehat) morphological operator to find light
# regions against a dark background (i.e., the credit card numbers)
tophat = cv2.morphologyEx(gray, cv2.MORPH_TOPHAT, rectKernel)
 Copy code 

Use our rectKernel And our grayscale images , We execute Top-hat Morphological operation , Store the results as tophat.

Top-hat Operate on a dark background （ Credit card number ） The light colored area is displayed below , As shown in the figure below ：

Given our high hat image , Let's calculate along x Directional gradient ：

# compute the Scharr gradient of the tophat image, then scale
# the rest back into the range [0, 255]
gradX = cv2.Sobel(tophat, ddepth=cv2.CV_32F, dx=1, dy=0,
	ksize=-1)
gradX = np.absolute(gradX)
(minVal, maxVal) = (np.min(gradX), np.max(gradX))
gradX = (255 * ((gradX - minVal) / (maxVal - minVal)))
gradX = gradX.astype("uint8")
 Copy code 

The next step in our effort to isolate the numbers is to calculate x High hat image in the direction Scharr gradient . Complete the calculation , Store the results as gradX .

In the calculation gradX After the absolute value of each element in the array , Let's take some steps to reduce the value to [0-255] Within the scope of （ Because the image is currently a floating point data type ）. So , We calculated gradX Of minVal and maxVal, Then calculate the second 73 The scaling equation shown in line （ That is the smallest / Maximum normalization ）. The last step is going to be gradX Convert to a range of [0-255] Of uint8. The results are shown in the following figure ：

Let's continue to improve the credit card number search algorithm ：

# apply a closing operation using the rectangular kernel to help
# cloes gaps in between credit card number digits, then apply
# Otsu's thresholding method to binarize the image
gradX = cv2.morphologyEx(gradX, cv2.MORPH_CLOSE, rectKernel)
thresh = cv2.threshold(gradX, 0, 255,
	cv2.THRESH_BINARY | cv2.THRESH_OTSU)[1]
# apply a second closing operation to the binary image, again
# to help close gaps between credit card number regions
thresh = cv2.morphologyEx(thresh, cv2.MORPH_CLOSE, sqKernel)
 Copy code 

To narrow the gap , We performed a close operation . Please note that , We used it again rectKernel. And then we went to gradX Image execution Otsu And binary threshold , Then there is another close operation . The results of these steps are as follows ：

Next, let's find the outline and initialize the digital grouping location list .

# find contours in the thresholded image, then initialize the
# list of digit locations
cnts = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL,
	cv2.CHAIN_APPROX_SIMPLE)
cnts = imutils.grab_contours(cnts)
locs = []
 Copy code 

We found the contours and stored them in a list cnts in . then , We initialize a list to hold the number group location .

Now let's traverse the contour , At the same time, filter according to the aspect ratio of each contour , Allows us to trim the number group position from other unrelated areas of the credit card ：

# loop over the contours
for (i, c) in enumerate(cnts):
	# compute the bounding box of the contour, then use the
	# bounding box coordinates to derive the aspect ratio
	(x, y, w, h) = cv2.boundingRect(c)
	ar = w / float(h)
	# since credit cards used a fixed size fonts with 4 groups
	# of 4 digits, we can prune potential contours based on the
	# aspect ratio
	if ar > 2.5 and ar < 4.0:
		# contours can further be pruned on minimum/maximum width
		# and height
		if (w > 40 and w < 55) and (h > 10 and h < 20):
			# append the bounding box region of the digits group
			# to our locations list
			locs.append((x, y, w, h))
 Copy code 

We loop through the contour in the same way as the reference image . After calculating the boundary rectangle of each contour c after , We calculate the aspect ratio by dividing the width by the height ar . Use aspect ratio , We analyze the shape of each contour . If ar Be situated between 2.5 and 4.0 Between （ Width greater than height ）, as well as 40 To 55 Between pixels w and 10 To 20 Between pixels h, We attach the boundary rectangle parameter in a convenient tuple to locs.

The following figure shows the groups we found —— For demonstration purposes , I let OpenCV A bounding box is drawn around each group ：

Next , We will sort the groups from left to right and initialize the list of credit card numbers ：

# sort the digit locations from left-to-right, then initialize the
# list of classified digits
locs = sorted(locs, key=lambda x:x[0])
output = []
 Copy code 

We according to the x It's worth it locs Sort , So they will be sorted from left to right . We initialize a list output , It will save the credit card number of the image . Now we know the position of each group of four digits , Let's loop through the four sorted groups and determine the numbers .

This cycle is quite long , It is divided into three code blocks —— This is the first block ：

# loop over the 4 groupings of 4 digits
for (i, (gX, gY, gW, gH)) in enumerate(locs):
	# initialize the list of group digits
	groupOutput = []
	# extract the group ROI of 4 digits from the grayscale image,
	# then apply thresholding to segment the digits from the
	# background of the credit card
	group = gray[gY - 5:gY + gH + 5, gX - 5:gX + gW + 5]
	group = cv2.threshold(group, 0, 255,
		cv2.THRESH_BINARY | cv2.THRESH_OTSU)[1]
	# detect the contours of each individual digit in the group,
	# then sort the digit contours from left to right
	digitCnts = cv2.findContours(group.copy(), cv2.RETR_EXTERNAL,
		cv2.CHAIN_APPROX_SIMPLE)
	digitCnts = imutils.grab_contours(digitCnts)
	digitCnts = contours.sort_contours(digitCnts,
		method="left-to-right")[0]
 Copy code 

In the first block of this loop , We extracted and filled groups on each side 5 Pixel , Apply threshold processing , And find and sort profiles . Please refer to the code for details . The extracted individual groups are shown below ：

Let's continue the loop with nested loop for template matching and similarity score extraction ：

# loop over the digit contours
	for c in digitCnts:
		# compute the bounding box of the individual digit, extract
		# the digit, and resize it to have the same fixed size as
		# the reference OCR-A images
		(x, y, w, h) = cv2.boundingRect(c)
		roi = group[y:y + h, x:x + w]
		roi = cv2.resize(roi, (57, 88))
		# initialize a list of template matching scores 
		scores = []
		# loop over the reference digit name and digit ROI
		for (digit, digitROI) in digits.items():
			# apply correlation-based template matching, take the
			# score, and update the scores list
			result = cv2.matchTemplate(roi, digitROI,
				cv2.TM_CCOEFF)
			(_, score, _, _) = cv2.minMaxLoc(result)
			scores.append(score)
		# the classification for the digit ROI will be the reference
		# digit name with the *largest* template matching score
		groupOutput.append(str(np.argmax(scores)))
 Copy code 

Use cv2.boundingRect We get the extract that contains each number ROI Required parameters . In order to make template matching work with some degree of accuracy , We will roi The size is adjusted with us in the 144 Reference on line OCR-A Font digital image （57×88 Pixels ） The same size .

We initialized a score list . Consider it our confidence score —— It is higher , The more likely it is to be the right template .

Now? , Let's cycle through each reference number （ The third nested loop ） And perform template matching . This is where the heavy work is done for this script .

OpenCV There is one named cv2.matchTemplate Convenient function of , You can provide two images ： One is the template , The other is the input image . take cv2.matchTemplate The purpose of applying to these two images is to determine their similarity .

under these circumstances , We provide reference digitROI Image and credit card with candidate numbers roi. Use these two images , We call the template matching function and store the results . Next , We extract scores from the results and append them to our score list . This completes the innermost cycle .

Use score （ Every number 0-9 One ）, We take the maximum score —— The maximum score should be the number we correctly identify . We found the number with the highest score , adopt np.argmax Get a specific index . The integer name of the index represents the most likely number based on the comparison with each template （ Remember again , The index has been pre sorted as 0-9）.

Last , Let's draw a rectangle around each group , And view the credit card number on the image in red text ：

# draw the digit classifications around the group
	cv2.rectangle(image, (gX - 5, gY - 5),
		(gX + gW + 5, gY + gH + 5), (0, 0, 255), 2)
	cv2.putText(image, "".join(groupOutput), (gX, gY - 15),
		cv2.FONT_HERSHEY_SIMPLEX, 0.65, (0, 0, 255), 2)
	# update the output digits list
	output.extend(groupOutput)
 Copy code 

For the third and last block of this loop , Let's draw a... Around the group 5 Pixel fill rectangle , Then draw text on the screen .

The last step is to append the number to the output list . Pythonic The method is to use the extension function to set the iteratable object （ In this case, list ） Each element of is appended to the end of the list .

To view the execution of the script , Let's output the result to the terminal and display our image on the screen .

# display the output credit card information to the screen
print("Credit Card Type: {}".format(FIRST_NUMBER[output[0]]))
print("Credit Card #: {}".format("".join(output)))
cv2.imshow("Image", image)
cv2.waitKey(0)
 Copy code 

Print the credit card type to the console , Then on the following day 173 The bank prints the credit card number .

In the last few lines , We display the image on the screen and wait for any key to be pressed , Then exit the script 174 and 175 That's ok .

Take time to congratulate yourself —— You did . Take a look back. （ At a high level ）, This script ：

• Store credit card types in a dictionary .
• Obtain the reference image and extract the digital image .
• Store the number template in the dictionary .
• Localize four credit card number groups , Each group contains four digits （ in total 16 Digit number ）.
• Extract to “ matching ” The number of .
• Perform template matching for each number , Put each individual ROI With each digital template 0-9 Compare , Store the score of each attempt to match at the same time .
• Find the highest score for each candidate number , And build a project called output A list of , It contains the credit card number .
• Output the credit card number and credit card type to our terminal , And display the output image on our screen .

Now it's time to look at the running script and check our results .

The credit card OCR result

Now we have credit cards OCR The system is coded , Let's try .

We obviously can't use a real credit card number in this example , So I used Google to collect some sample images of credit cards .

These credit cards are obviously fake , For demonstration purposes only . however , You can apply the same techniques in this blog post to identify the numbers on the actual credit card .

To check our credit cards OCR The operation of the system , Please open a terminal and execute the following command ：

$ python ocr_template_match.py --reference ocr_a_reference.png \
	--image images/credit_card_05.png
Credit Card Type: MasterCard
Credit Card #: 5476767898765432
 Copy code 

Our first result image ,100% correct ：

Please note how we can correctly mark credit cards as MasterCard , Just check the first digit in the credit card number . Let's try the second picture , This is a visa ：

$ python ocr_template_match.py --reference ocr_a_reference.png \
	--image images/credit_card_01.png
Credit Card Type: Visa
Credit Card #: 4000123456789010
 Copy code 

Once again, , We can correctly match credit cards using template matching OCR.

$ python ocr_template_match.py --reference ocr_a_reference.png \
	--image images/credit_card_02.png
Credit Card Type: Visa
Credit Card #: 4020340002345678
 Copy code 

summary

In this tutorial , We learned how to pass OpenCV and Python Perform optical character recognition using template matching (OCR).

say concretely , We apply our template matching OCR Method to identify the type of credit card and 16 A credit card number .

To achieve this , We divide the image processing pipeline into 4 A step ：

• Detect four groups of four numbers on the credit card through various image processing technologies , Including morphological operations 、 Threshold and contour extraction .
• Extract each individual number from the four groups , Lead to the need for classification 16 A digital .
• By associating each number with OCR-A Compare Fonts , Apply template matching to each number to get our number classification .
• Check the first digit of the credit card number to determine the issuing company .

After evaluating our credit cards OCR After the system , We found it was 100% accurate , Provided that the issuing credit card company uses OCR-A Font as number .

To extend this application , You may want to collect real images of credit cards in the field , And may train machine learning models （ Through standard feature extraction or training or convolution neural network ） To further improve the accuracy of the system .

I hope you like this article about OCR 's blog post .