Apparently something about computer vision projects is irresistible to me. A job post appeared on Upwork the other day requesting help with automatically reading out gauges from images of a dashboard. There were two sample images attached so I immediately started playing around with how one might accomplish this. The images look like this, with some variation in image size, camera angle, lighting and image quality:
%matplotlib inline
import cv2
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
data_path = "./data/"
image = cv2.imread(data_path + "sample1b.jpg")
fig, ax = plt.subplots(figsize=(6, 6))
ax.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB)) # Matplotlib imshow functions expects RGB, opencv defaults to BGR
General Approach¶
My first thought was that the red "TEMP COMP" rectangle would be pretty easy to pick out. If the images were all the same resolution then we could also easily locate the gauge centers since they would always be the same vector displacements away from the center of the red rectangle, after accounting for image rotation (based on the rectangle angle). Sadly the images don't all have the same resolution; but the gauge centers are pretty obvious to a human eye and might be detected with Circle/Blob detection on their own.
Once you know the gauge centers, and have properly rotated the coordinate system so that you know which pixels are the "top" or zero-reading of the gauges, how do you identify where the gauge needles are pointing? My first idea was to draw a large circle centered at the gauge center and pull pixel values along the circle perimeter; where the perimeter crosses the needle there will be a cluster of very dark pixel values. My second (and better) idea was to draw radial lines out from the center of the gauge and pull pixel values on each line; when a line lies along a needle all the pixel values will be relatively tightly clustered around black, whereas other lines will start out black and then transition to white.
Below I walk through an implementation for all these different parts of the pipeline.
Task 1: Red Rectangle Detection¶
The red "TEMP COMP" rectangle will allow us to correct for image rotation so we know where the true zero-reading (top) of each gauge is. Very conveniently this is the only significantly red region, and the only rectangle in this size range!
Color masking is most effective in HSV space, but unfortunately "red" has values which wrap around in the Hue dimension; Hue between 0-10 and 170-180
img_hsv = cv2.cvtColor(image,cv2.COLOR_BGR2HSV) # Convert to HSV space
# Lower hues mask (0-10)
lower_red, upper_red = np.array([0,50,20]), np.array([10,255,255])
mask0 = cv2.inRange(img_hsv, lower_red, upper_red)
# Upper hues mask (170-180)
lower_red, upper_red = np.array([170,50,0]), np.array([180,255,255])
mask1 = cv2.inRange(img_hsv, lower_red, upper_red)
# Combine masks
mask = mask0 + mask1
fig, ax = plt.subplots(figsize=(7, 7))
ax.set_title("Red Color Thresholded Mask", fontsize=16)
ax.imshow(mask, cmap="Greys")
For downstream processing we may want the average brighness of this red rectangular region to use in adjusting other threshold values. That is straightforward to calculate using an inverted version of this mask and the cv2.mean() function.
channel_means = cv2.mean(img_hsv, mask=~mask) # Invert mask to sample only the rectangle
print(channel_means)
print("Mean value (brightness) in red rectangle is %.2f" % (channel_means[2],))
We could pass the mask to the Canny() edge detector now and expect it to do a bang up job finding the rectangle edges (as well as the internal text edges). A Gaussian blur is a common step in edge detection in order to degrade anything that is "edge-like" but not really an edge; it's not really necessary here so we would pass a small aperture size (kernel size) when using Canny.
Edge detection is a common preprocessing step to get a clean form of input to pass into a contour detector. A contour is a closed curve of pixels with the same color or intensity forming a continuous boundary of an object, so they are a very useful tool for shape or object detection. (Note, in opencv the convention for contours is that the object is white and background is black.) More on edges vs. contours from this stackoverflow discussion:
Edges are computed as points that are extrema of the image gradient in the direction of the gradient...edge pixels are a local notion: they just point out a significant difference between neighbouring pixels...Contours are often obtained from edges, but they are aimed at being object contours. Thus, they need to be closed curves. You can think of them as boundaries...
In this case our rectangle shape is already very clean, so we will just pass the binary mask obtained from the color range operation directly to a contour finder.
# RETR_EXTERNAL is the contour retrieval mode, CHAIN_APPROX_NONE is the contour approximation method
# Output is a Python list of all the found contours, each one is a Numpy array of (x,y) coordinates
(_,contours,_) = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
# For each contour check object area with a bounding box, if area above threshold assume we found the rectangle
imcopy = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).copy()
for contour in contours:
(x,y,w,h) = cv2.boundingRect(contour)
# A threshold area of 1000 works for all the images in the data set (which can have different resolutions)
if w*h > 1000:
rect = cv2.minAreaRect(contour) # Fit a rotated rectangle bounding box to the contour
print("Rotated rectangle found at center point" , np.round(rect[0]), "with angle", np.round(rect[2], 2), "degrees.")
print("Rotated rectangle width is %i and area is %i" % (np.round(min(rect[1][0], rect[1][1])), np.round(rect[1][0]*rect[1][1])))
rotrect_angle, rect_width, rect_area = rect[2], min(rect[1][0], rect[1][1]), rect[1][0]*rect[1][1]
print(rect[0])
# Draw the rotated rectangle (see https://stackoverflow.com/questions/11779100/python-opencv-box2d)
box = cv2.boxPoints(rect) # cv2.boxPoints(rect) for OpenCV 3.x
box = np.int0(box)
cv2.drawContours(imcopy,[box], 0, (0,255,0), 1)
# See rotated rectangle superimposed on image
fig, ax = plt.subplots(figsize=(10, 10))
ax.set_title("Rotated Rectangle Bounding Box from Contours", fontsize=16)
ax.imshow(imcopy)
Rotated Rectangle Angle¶
Regarding the rotation angle output by the minAreaRect() function, it always exists on the half-closed interval [-90, 0), increasing from -90 to zero as the rectangle is rotated clockwise. A horizontal rectangle lying on it's longer side will have $\theta = -90$, as it rotates clockwise the angle increases towards zero and finally when the rectangle has been rotated into a completely "standing" vertical position the angle will tick back over again to $\theta = -90$. Likewise as the rectangle is rotated clockwise through the remaining two quadrants. For the "TEMP COMP" sign, when it is skewed slightly clockwise the angle is, for example, -88 degrees. When it is skewed slightly counterclockwise (as in the above example) the angle is, for example, -2 degrees.
However I find it most convenient to work in a conventional coordinate system where $\theta=0$ corresponds to the "TEMP COMP" rectangle exactly horizontal; going from 0 to 360 degrees as it is rotated fully clockwise or 0 to -360 degrees as it is rotated fully counterclockwise. Let's create a function that takes in the output angle from minAreaRect() and gives us back an angle in this more sensible convention. For this we need to assume that the image rotation is never more severe than 45 degrees off the horizontal.
def get_real_angle(rotrect_angle):
if rotrect_angle <= -45:
real_angle_deg = 90 + rotrect_angle
else:
real_angle_deg = rotrect_angle
return real_angle_deg
Functionalized Code¶
def find_rotated_rect(img_hsv, rect_area_thresh=1000):
# Lower hues mask (0-10)
lower_red, upper_red = np.array([0,50,50]), np.array([10,255,255])
mask0 = cv2.inRange(img_hsv, lower_red, upper_red)
# Upper hues mask (170-180)
lower_red, upper_red = np.array([170,50,50]), np.array([180,255,255])
mask1 = cv2.inRange(img_hsv, lower_red, upper_red)
# Combine masks
mask = mask0 + mask1
# RETR_EXTERNAL is the contour retrieval mode, CHAIN_APPROX_NONE is the contour approximation method
# Output is a Python list of all the found contours, each one is a Numpy array of (x,y) coordinates
(_,contours,_) = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
# For each contour check object area with a bounding box, if area above threshold assume we found the rectangle
imcopy = image.copy()
for contour in contours:
(x,y,w,h) = cv2.boundingRect(contour)
if w*h > rect_area_thresh:
rect = cv2.minAreaRect(contour) # Fit a rotated rectangle bounding box to the contour
print("Rotated rectangle found at center point" , np.round(rect[0]), "with angle", np.round(rect[2], 2), "degrees.")
print("Rotated rectangle area is" , np.round(rect[1][0]*rect[1][1]))
rotrect_angle, rect_width, rect_area = rect[2], min(rect[1][0], rect[1][1]), rect[1][0]*rect[1][1]
return rect, rotrect_angle, rect_area, rect_width
Task 2: Gauge Center Detection¶
Here we will first use color filtering to pull out black-ish pixels, then we will use Blob Detection with a circularity threshold and a pretty tight area threshold to find the central black knobs of the gauges. The "Value" AKA "Brightness" bound that we use for black is somewhat more critical than the red thresholding range used above; this is because there are a lot of other black regions, shadows and markings on the image in addition to the gauge center knobs. Ultimately we will address this by beginning at low bounding values and incrementing upward until we detect 6 blobs.
Since we are using blob area as a detection criteria, we would also like to scale that bounding value based on the area of the rectangular region. For reference image sample1b.jpg the red "TEMP COMP" rectangle has area of 8079, and using upper bound $V = 65$ we find the blobs have an average area of 118. So for example for sample2b.jpg with rectangle area of 13767 we would roughly expect blobs of area 118 * 13767/8079 = 201. To be safe we should set the minimum area criteria to maybe 80% of these values.
First lets walk through the steps with a suitable fixed upper bound on $V$, visualizing the intermediate processing.
# Black HSV mask
lower_black, upper_black = np.array([0, 0, 0]), np.array([180, 255, 65])
mask = cv2.inRange(img_hsv, lower_black, upper_black)
fig, ax = plt.subplots(figsize=(7, 7))
ax.set_title("Black Color Thresholded Mask", fontsize=16)
ax.imshow(mask, cmap="Greys")
Here a blur will help us to eliminate some of the noise from the black gauge lettering; the circular central regions of the gauges will be less affected by the blurring than the narrow needle regions because the majority of their neighboring pixels are also dark, which will improve the circularity of our blobs. Also for blob detection we want dark blobs on a light background so we will bitwise NOT the mask. Since the mask is dtype uint8 the ~ operator will invert values on the range [0, 255].
# Apply a Gaussian Blur to the inverted mask
blurred = cv2.blur(~mask, (9, 9))
fig, ax = plt.subplots(figsize=(7, 7))
ax.set_title("Blurred and Inverted Mask", fontsize=16)
ax.imshow(blurred)
In the opencv blob detector, the threshold parameters are used to determine the thresholding range over which the "repeatability" of detection for a blob is assessed. Repeatability means how stable a blob is with respect to setting different thresholds in the grayscale image (see this stackoverflow answer). For a given threshold, pixel values below are set to 0 and pixel values above are set to 1, and the resulting binary is analyzed for blobs. If the same blob is found using multiple different threshold settings in the specified range, then it is a stable blob. The minRepeatability parameter sets the required number of times a blob should be detected to be considered stable (so a value of 1 does not require any repeatability). In this case we expect the inner part of the knobs to appear as very dark blobs, even after blurring, so we will keep the minThreshold and maxThreshold low to zero in on these objects rather than e.g. the metal screws on the far left and right which secure the panel and show up as more lightly colored blobs.
# Setup SimpleBlobDetector parameters.
params = cv2.SimpleBlobDetector_Params()
# Adjust parameters for determining blob stability/repeatability
params.minThreshold = 0;
params.maxThreshold = 100;
params.minRepeatability = 1
# Filter by Area.
params.filterByArea = True
params.minArea = 0.75 * 118 * rect_area / 8079 # rect_area is found above during red rectangle detection
# Filter by Circularity
# params.filterByCircularity = True
# params.minCircularity = 0.5
# Create a detector with the parameters
detector = cv2.SimpleBlobDetector_create(params)
# Detect blobs
keypoints = detector.detect(blurred)
# Draw detected blobs as green circles on the blurred mask
imcopy = image.copy()
im_with_keypoints = cv2.drawKeypoints(blurred, keypoints,
np.array([]), (0,255,0),
cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS) # ensures circle size corresponds to blob size
fig, ax = plt.subplots(figsize=(6, 6))
ax.set_title("Detected Blobs in Blurred Mask", fontsize=16)
ax.imshow(im_with_keypoints)
Now let's wrap this in a while loop that will start at lower $V$ values and increment upward until we find 6 blobs.
image = cv2.imread(data_path + "sample1b.jpg")
img_hsv = cv2.cvtColor(image,cv2.COLOR_BGR2HSV) # Convert to HSV space
# Increment upper bound for "value" (brightness) until we find 6 blobs
n_blobs = 0
v_upper = 25
while n_blobs < 6 and v_upper < 100:
# Black HSV mask
lower_black, upper_black = np.array([0, 0, 0]), np.array([180, 255, v_upper])
mask = cv2.inRange(img_hsv, lower_black, upper_black)
# Apply a Gaussian Blur to the inverted mask
blurred = cv2.blur(~mask, (9, 9))
# Detect blobs
keypoints = detector.detect(blurred)
n_blobs = len(keypoints)
v_upper += 5
# Draw detected blobs as green circles on the original image
imcopy = image.copy()
im_with_keypoints = cv2.drawKeypoints(imcopy, keypoints,
np.array([]), (0,255,0),
cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS) # ensures circle size corresponds to blob size
fig, ax = plt.subplots(figsize=(6, 6))
ax.set_title("Detected Blobs in Blurred Mask", fontsize=16)
ax.imshow(im_with_keypoints)
From the identified blobs we extract the indices for the center points of the six gauges in the (column index, row index) convention. We take the gauges as numbered 1 through 6 starting with the leftmost in the top row and ending with the rightmost in the bottom row, and we can sort the gauge center indices to put them in this order.
# Extract coordinates of gauge centers in the (column index, row index) convention
gauge_center_coords = []
for kp in keypoints:
gauge_center_coords.append((np.int(kp.pt[0]), np.int(kp.pt[1])))
# Sort the gauge's by center coordinates
gauge_center_coords.sort(key=lambda x: x[1])
gauge_center_coords[0:4] = sorted(gauge_center_coords[0:4], key=lambda x: x[0])
gauge_center_coords[4:] = sorted(gauge_center_coords[4:], key=lambda x: x[0])
print("Gauge centers found at coordinates: ", gauge_center_coords)
Functionalized Code¶
def find_gauge_centers(img_hsv, rect_area):
# Setup SimpleBlobDetector parameters.
params = cv2.SimpleBlobDetector_Params()
# Adjust parameters for determining blob stability/repeatability
params.minThreshold = 0;
params.maxThreshold = 100;
params.minRepeatability = 1
# Filter by Area.
params.filterByArea = True
params.minArea = 0.75 * 118 * rect_area / 8079 # rect_area is found above during red rectangle detection
# Filter by Circularity.
# params.filterByCircularity = True
# params.minCircularity = 0.5
detector = cv2.SimpleBlobDetector_create(params)
# Increment upper bound for "value" (brightness) until we find 6 blobs
n_blobs = 0
v_upper = 25
while n_blobs < 6:
# Black HSV mask
lower_black, upper_black = np.array([0, 0, 0]), np.array([180, 255, v_upper])
mask = cv2.inRange(img_hsv, lower_black, upper_black)
# Apply a Gaussian Blur to the inverted mask
blurred = cv2.blur(~mask, (9, 9))
# Detect blobs
keypoints = detector.detect(blurred)
n_blobs = len(keypoints)
v_upper += 5
# Extract the coordinates of gauge centers in the (column index, row index) convention
gauge_center_coords = []
for kp in keypoints:
gauge_center_coords.append((np.int(kp.pt[0]), np.int(kp.pt[1])))
# Sort the gauge's by center coordinates
gauge_center_coords.sort(key=lambda x: x[1])
gauge_center_coords[0:4] = sorted(gauge_center_coords[0:4], key=lambda x: x[0])
gauge_center_coords[4:] = sorted(gauge_center_coords[4:], key=lambda x: x[0])
print("Gauge centers found at coordinates: ", gauge_center_coords)
return keypoints, gauge_center_coords
Task 3: Locating Gauge Needles¶
Now we'd like to draw circles of roughly the same size as each gauge, then construct radial lines out from the center of the gauge to each perimeter pixel on the circle. For each of these lines we need to know what angle it makes on the gauge's face (which means accounting for the image rotation) and also what the pixel values are along that line (lines lying along the gauge needle will have uniformly black pixel values).
The size of the circle needed to encompass a gauge's face will depend on the resolution of the image. Luckily the size of our "TEMP COMP" rectangle gives us a way to scale this size correctly. For our sample1b reference image the rectangle had a width of 57 pixels and a good circle radius size was 30 pixels.
For a single gauge, the steps are:
- draw a circle about the gauge center point and retrieve the center and perimeter pixel locations in (row, col) index convention
- reverse the row index values so that we have a conventional frame of reference with "origin" (0, 0) in lower left rather than upper left
- convert pixel locations to (x, y) coordinate convention rather than (row, col) convention
- use trigonometric equations to rotate the pixel location vectors about the center of the rectangle
- identify the "topmost" pixel on the gauge perimeter in the rotated coordinate system; this pixel is at "clock angle" zero i.e. 12 o'clock
- draw radial lines from gauge center to perimeter pixels
- extract grayscale pixel values along each line (and their mean and standard deviation)
- compute the angle between each line and the 12 o'clock line
- identify the needle angle as the clock angle for the line with the darkest average pixel value
- convert the needle angle into a reading appropriately (some gauges read CW, some CCW)
I walk through these steps below.
image = cv2.imread(data_path + "sample1b.jpg")
img_hsv = cv2.cvtColor(image,cv2.COLOR_BGR2HSV) # Convert to HSV space
# Pick out the first gauge to work on
circ_col, circ_row = gauge_center_coords[0]
# Extract rotated rectangle parameters
rotrect_angle, rect_width, rect_area = rect[2], min(rect[1][0], rect[1][1]), rect[1][0]*rect[1][1]
rect_col, rect_row = rect[0]
# Determine good size for circles to be drawn around gauges
circ_rad = np.int(30 * rect_width / 57)
# Draw a circle centered around gauge center point, extract pixel indices and colors on circle perimeter
blank = np.zeros(image.shape[:2], dtype=np.uint8)
cv2.circle(blank, (circ_col, circ_row), circ_rad, 255, thickness=1) # Draw function wants center point in (col, row) order like coordinates
ind_row, ind_col = np.nonzero(blank)
b = image[:, :, 0][ind_row, ind_col]
g = image[:, :, 1][ind_row, ind_col]
r = image[:, :, 2][ind_row, ind_col]
colors = list(zip(b, g, r))
# "reverse" the row indices to get a right-handed frame of reference with origin in bottom left of image
ind_row_rev = [image.shape[0] - row for row in ind_row]
circ_row_rev = image.shape[0] - circ_row
rect_row_rev = image.shape[0] - rect_row
# Convert from indexes in (row, col) order to coordinates in (col, row) order
circ_x, circ_y = circ_col, circ_row_rev
original_coord = list(zip(ind_col, ind_row_rev))
rect_x, rect_y = rect_col, rect_row_rev
# Rotate coords about rectangle center in order to identify topmost pixel of gauges
temp_x, temp_y = [x - rect_x for x in ind_col], [y - rect_y for y in ind_row_rev] # Translate from rectangle center point
angle_deg = get_real_angle(rotrect_angle)
theta = angle_deg * (np.pi/180)
rotated = []
for (x, y) in list(zip(temp_x, temp_y)):
rotated.append(((x*np.cos(theta) - y*np.sin(theta)) + rect_x,
(y*np.cos(theta) + x*np.sin(theta)) + rect_y)) # Rotate about 0,0 then reverse translation from rectangle center point
top_yval = max([y for (x,y) in rotated])
top_pixel = [(x, y) for (x, y) in rotated if y == top_yval][0]
# Translate coords from gauge centers in order to compute angle between points on the perimeter
translated = []
for (x, y) in original_coord:
translated.append((x - circ_x, y - circ_y))
# Construct dataframe holding various coordinate representations and pixel values
df = pd.DataFrame({"indices":list(zip(ind_col, ind_row)), "orig":original_coord, "rot": rotated, "trans": translated, "color": colors})
# Identify the pixel which is the topmost point of the circle when properly rotated
df["top_pixel"] = (df["rot"] == top_pixel)
top_trans_pix = df.loc[df["top_pixel"], "trans"].values[0]
df.head()
# Visualize the circle and topmost circle pixel
imcopy = image.copy()
fig, ax = plt.subplots(figsize=(18, 6))
ax.set_title("Sample Perimeter Circle and Topmost Pixel", fontsize=16)
cv2.circle(imcopy, (circ_col, circ_row), circ_rad, 255, thickness=1) # Draw circle around gauge center point
top_orig_pix = df.loc[df["top_pixel"], "indices"].values[0] # Get indices for "topmost" pixel on circle after rotation
cv2.circle(imcopy, top_orig_pix, 1, 255, thickness=3) # Draw topmost pixel
ax.imshow(imcopy)
# Angle and "clock angle" between the topmost pixel and other perimeter pixels
angles = []
for vec in df["trans"].values:
angles.append((180 / np.pi) * np.arccos(np.dot(top_trans_pix, vec) / (np.linalg.norm(top_trans_pix) * np.linalg.norm(vec))))
df["angle"] = angles
df["clock_angle"] = df["angle"] + (-2*df["angle"] + 360)*(df["trans"].apply(lambda x: x[0] < 0)).astype(int)
# Draw lines between gauge center and perimeter pixels and compute mean and std dev of pixels along lines
stds = []
means = []
gray_values = []
for (pt_col, pt_row_rev) in df["orig"].values:
pt_row = -(pt_row_rev - image.shape[0])
blank = np.zeros(image.shape[:2], dtype=np.uint8)
cv2.line(blank, (circ_col, circ_row), (pt_col, pt_row), 255, thickness=2) # Draw function wants center point in (col, row) order like coordinates
ind_row, ind_col = np.nonzero(blank)
b = image[:, :, 0][ind_row, ind_col]
g = image[:, :, 1][ind_row, ind_col]
r = image[:, :, 2][ind_row, ind_col]
grays = (b.astype(int) + g.astype(int) + r.astype(int))/3 # Compute grayscale with naive equation
stds.append(np.std(grays))
means.append(np.mean(grays))
gray_values.append(grays)
df["stds"] = stds
df["means"] = means
df["gray_values"] = gray_values
# Visualize a sample of the radial lines
imcopy = image.copy()
fig, ax = plt.subplots(figsize=(18, 6))
ax.set_title("Sample Radial Lines", fontsize=16)
# Draw every fifth radial line
for (pt_col, pt_row_rev) in df["orig"].values[::5]:
pt_row = -(pt_row_rev - image.shape[0])
cv2.line(imcopy, (circ_col, circ_row), (pt_col, pt_row), 255, thickness=1) # Draw function wants center point in (col, row) order like coordinates
cv2.circle(imcopy, (circ_col, circ_row), circ_rad, 255, thickness=1) # Draw circle around gauge center point
ax.imshow(imcopy[0:100, 0:120])
# Plot mean pixel value as a function of needle "clock angle" (zero degrees is 12 o'clock)
fig, ax = plt.subplots()
ax2 = ax.twinx()
ax2.scatter(df["clock_angle"], df["stds"], color="r", alpha=0.3, label="pixel std. dev.")
ax.scatter(df["clock_angle"], df["means"], label="pixel mean", color="b", alpha=0.3)
ax2.legend(loc="lower center")
ax.legend(loc="lower left")
ax.set_xlabel("Clock Angle of Radial Line")
ax.set_ylabel("Metric Value along Radial Line")
ax.set_title("Locating Gauge Needle from Radial Line Pixel Values", fontsize=16)
# What angle gives us the minimum in mean pixel value (i.e. darkest average of pixel intensity)
min_mean = df["means"].min()
needle_angle = df.loc[df["means"] == min_mean, "clock_angle"].values[0]
print("Darkest average pixel value (i.e. the needle) is found at clock angle %.2f degrees" % (needle_angle,))
When looking at either the standard deviation or the mean value of grayscale pixel intensities along the different radial lines we can very clearly identify the presence of the gauge needle as a sharp minimum in the metrics at around 330 degrees. A more nuanced metric is also the "bimodality" of the pixel value histogram. For instance if we compare the radial line where we find the needle to one around 158 degrees:
# Plot histogram of pixel values for needle line and non-needle line
fig, ax = plt.subplots()
grays = df.loc[df["means"].values == min_mean, "gray_values"].values
ax.hist(grays, alpha=0.45, label="Needle Line")
grays = df.loc[np.round(df["clock_angle"].values) == 158, "gray_values"].values
ax.hist(grays, alpha=0.45, label="Non-needle Line")
ax.legend()
ax.set_xlabel("Grayscale Pixel Value")
ax.set_ylabel("Count")
ax.set_title("Distribution of Pixel Values", fontsize=16)
# Visualize the "needle" radial lines
imcopy = image.copy()
fig, ax = plt.subplots(figsize=(18, 6))
ax.set_title("Radial Line Identified as Needle Location", fontsize=16)
(pt_col, pt_row) = df.loc[df["means"] == min_mean, "indices"].values[0]
cv2.line(imcopy, (circ_col, circ_row), (pt_col, pt_row), (0, 255, 0), thickness=1) # Draw needle radial line
cv2.circle(imcopy, (circ_col, circ_row), circ_rad, 255, thickness=1) # Draw circle around gauge center point
ax.imshow(imcopy[0:100, 0:120])
Once the needle angle has been determined, the final step is to convert the angle into a reading based on the read-out convention for the particular gauge. Gauges 2 and 4 read clockwise while the other gauges read counterclockwise.
readout_conventions = ["CCW", "CW", "CCW", "CW", "CW", "CW"]
convention = readout_conventions[0] # Counterclockwise readout for Gauge #1
if convention == "CW":
print("Gauge #1 reading is %.2f" % (10*needle_angle/360,))
else:
print("Gauge #1 reading is %.2f" % (10 - (10*needle_angle/360),))
Functionalized Code¶
def find_needle(image, circ_col, circ_row, rect):
# Extract rotated rectangle parameters
rotrect_angle, rect_width, rect_area = rect[2], min(rect[1][0], rect[1][1]), rect[1][0]*rect[1][1]
rect_col, rect_row = rect[0]
# Determine good size for circles to be drawn around gauges
circ_rad = np.int(30 * rect_width / 57)
# Draw circle centered around gauge center point, extract pixel indices and colors on circle perimeter
blank = np.zeros(image.shape[:2], dtype=np.uint8)
cv2.circle(blank, (circ_col, circ_row), circ_rad, 255, thickness=1) # Draw function wants center point in (col, row) order like coordinates
ind_row, ind_col = np.nonzero(blank)
b = image[:, :, 0][ind_row, ind_col]
g = image[:, :, 1][ind_row, ind_col]
r = image[:, :, 2][ind_row, ind_col]
colors = list(zip(b, g, r))
# "reverse" the row indices to get a right-handed frame of reference with origin in bottom left of image
ind_row_rev = [image.shape[0] - row for row in ind_row]
circ_row_rev = image.shape[0] - circ_row
rect_row_rev = image.shape[0] - rect_row
# Convert from indexes in (row, col) order to coordinates in (col, row) order
circ_x, circ_y = circ_col, circ_row_rev
original_coord = list(zip(ind_col, ind_row_rev))
rect_x, rect_y = rect_col, rect_row_rev
# Rotate coords about rectangle center in order to identify topmost pixel of gauges
temp_x, temp_y = [x - rect_x for x in ind_col], [y - rect_y for y in ind_row_rev] # Translate from rectangle center point
angle_deg = get_real_angle(rotrect_angle)
theta = angle_deg * (np.pi/180)
rotated = []
for (x, y) in list(zip(temp_x, temp_y)):
rotated.append(((x*np.cos(theta) - y*np.sin(theta)) + rect_x,
(y*np.cos(theta) + x*np.sin(theta)) + rect_y)) # Rotate about 0,0 then reverse translation from rectangle center point
top_yval = max([y for (x,y) in rotated])
top_pixel = [(x, y) for (x, y) in rotated if y == top_yval][0]
# Translate coords from gauge centers in order to compute angle between points on the perimeter
translated = []
for (x, y) in original_coord:
translated.append((x - circ_x, y - circ_y))
# Construct dataframe holding various coordinate representations and pixel values
df = pd.DataFrame({"indices":list(zip(ind_col, ind_row)), "orig":original_coord, "rot": rotated, "trans": translated, "color": colors})
# Identify the pixel which is the topmost point of the circle when properly rotated
df["top_pixel"] = (df["rot"] == top_pixel)
top_trans_pix = df.loc[df["top_pixel"], "trans"].values[0]
# Angle and "clock angle" between the topmost pixel and other perimeter pixels
angles = []
for vec in df["trans"].values:
angles.append((180 / np.pi) * np.arccos(np.dot(top_trans_pix, vec) / (np.linalg.norm(top_trans_pix) * np.linalg.norm(vec))))
df["angle"] = angles
df["clock_angle"] = df["angle"] + (-2*df["angle"] + 360)*(df["trans"].apply(lambda x: x[0] < 0)).astype(int)
# Draw lines between gauge center and perimeter pixels and compute mean and std dev of pixels along lines
stds = []
means = []
gray_values = []
for (pt_col, pt_row_rev) in df["orig"].values:
pt_row = -(pt_row_rev - image.shape[0])
blank = np.zeros(image.shape[:2], dtype=np.uint8)
cv2.line(blank, (circ_col, circ_row), (pt_col, pt_row), 255, thickness=2) # Draw function wants center point in (col, row) order like coordinates
ind_row, ind_col = np.nonzero(blank)
b = image[:, :, 0][ind_row, ind_col]
g = image[:, :, 1][ind_row, ind_col]
r = image[:, :, 2][ind_row, ind_col]
grays = (b.astype(int) + g.astype(int) + r.astype(int))/3 # Compute grayscale with naive equation
stds.append(np.std(grays))
means.append(np.mean(grays))
gray_values.append(grays)
df["stds"] = stds
df["means"] = means
df["gray_values"] = gray_values
# Find needle clock angle
min_mean = df["means"].min()
needle_angle = df.loc[df["means"] == min_mean, "clock_angle"].values[0] # Find needle angle
return df, needle_angle
def read_gauge(angle, convention):
# Gauge readout according to convention
if convention == "CW":
readout = 10*needle_angle/360
else:
readout = 10 - (10*needle_angle/360)
return readout
Final Results¶
Here are the results of the full pipeline on the two example images provided. Not bad!
readout_conventions = ["CCW", "CW", "CCW", "CW", "CW", "CW"]
image = cv2.imread(data_path + "sample1b.jpg")
imcopy = image.copy()
img_hsv = cv2.cvtColor(image,cv2.COLOR_BGR2HSV) # Convert to HSV space
# Find rectangle panel and gauge centers
rect, rotrect_angle, rect_area, rect_width = find_rotated_rect(img_hsv, rect_area_thresh=1000)
keypoints, gauge_centers = find_gauge_centers(img_hsv, rect_area)
# For each gauge, get gauge readout and visualize results
for i, ((c, r), convention) in enumerate(zip(gauge_centers, readout_conventions)):
# Find needle angle
df, needle_angle = find_needle(image=image, circ_col=c, circ_row=r, rect=rect)
# Draw needle radial line and gauge center point
(pt_col, pt_row) = df.loc[df["clock_angle"] == needle_angle, "indices"].values[0]
cv2.line(imcopy, (c, r), (pt_col, pt_row), (0, 255, 0), thickness=1) # Draw needle radial line
cv2.circle(imcopy, (c, r), 1, (0, 255, 0), thickness=3) # Draw function wants center point in (col, row) order like coordinates
# Gauge readout according to convention
print("Gauge #%i reading is %.2f" % (i+1, read_gauge(needle_angle, convention)))
# Draw the rotated rectangle (see https://stackoverflow.com/questions/11779100/python-opencv-box2d)
cv2.drawContours(imcopy,[np.int0(cv2.boxPoints(rect))], 0, (0,255,0), 1)
fig, ax = plt.subplots(figsize=(18, 6))
ax.imshow(imcopy)
image = cv2.imread(data_path + "sample2b.jpg")
imcopy = image.copy()
img_hsv = cv2.cvtColor(image,cv2.COLOR_BGR2HSV) # Convert to HSV space
rect, rotrect_angle, rect_area, rect_width = find_rotated_rect(img_hsv, rect_area_thresh=1000)
keypoints, gauge_centers = find_gauge_centers(img_hsv, rect_area)
# For each gauge, find needle reading
for i, ((c, r), readout) in enumerate(zip(gauge_centers, readout_conventions)):
df, needle_angle = find_needle(image=image, circ_col=c, circ_row=r, rect=rect)
# Draw needle radial line and gauge center point
(pt_col, pt_row) = df.loc[df["clock_angle"] == needle_angle, "indices"].values[0]
cv2.line(imcopy, (c, r), (pt_col, pt_row), (0, 255, 0), thickness=1) # Draw needle radial line
cv2.circle(imcopy, (c, r), 1, (0, 255, 0), thickness=3) # Draw function wants center point in (col, row) order like coordinates
# Gauge readout according to convention
print("Gauge #%i reading is %.2f" % (i+1, read_gauge(needle_angle, convention)))
# Draw the rotated rectangle (see https://stackoverflow.com/questions/11779100/python-opencv-box2d)
cv2.drawContours(imcopy,[np.int0(cv2.boxPoints(rect))], 0, (0,255,0), 1)
fig, ax = plt.subplots(figsize=(18, 6))
ax.imshow(imcopy)
