The goals of this project are as follows:
This is project #4 of my Self Driving Car Projects
Python source files:
lib/calibrate.py- calibrate camera from images and save calibration matrix and distortion coefficients to
lib/camera.py- calibrated camera interface
lib/draw.py- drawing functions
lib/lane_detection.py- lane detection
lib/line_fit.py- line fit and detection pipeline
test_camera.py- write images from running
camera.pyto calibration images
run.py- process video by running
line_fit.pyfor each image in video if video is provided, otherwise run
process_imageand detection function outputs on test images.
camera_cal/- chessboard images to calibrate camera.
output_images/calib_corners/- corners drawn on chessboard images using calibrated camera matrix
output_images/test_images/- undistorted images using calibrated camera matrix on test images
output_images/test_images-unwarped/- unwarped detection output on test images
output_images/test_images-warped/- warped detection output on test images
output_images/test_images-pipeline/- pipeline output on test images
Before we start with the pipeline that processes the images in the video, we need to calibrate the camera. This is needed since the images from the camera are usually distorted due to its lense curvature and other reasons. Calibration here means we want to determine the camera matrix and distortion coefficients that can be used to undistort the images. This is done by feeding chessboard images to
calibrate.py to obtain all chessboard corners in the image. These corner coordinates in the image are referred to as image points.
These image corner coordinates are mapped to their corresponding real world coordinates, which are [[0,0,0], [1,0,0], …, [8,5,0]] and referred to as object points. This mapping will be feed to
cv2.calibrateCamera to return the camera calibration matrix and distortion coefficients needed.
Here is a sample output of corners detected and drawn:
Chessboard Corners Drawn
Here is a different chessboard image after applying
cv2.undistort function with the obtained camera calibration matrix and distortion coefficients:
Undistorted Chessboard Image
Here is the original distorted image for comparison.
Distorted Chessboard Input Image
Here is the undistortion applied to a road image.
Sample undistorted lane image
Original distorted lane image
To extract lane lines, a combination of color and gradient thresholds are used to filter lane line pixels. We then take these lane line pixels to generate a binary image needed for the next step.
The yellow lines are identified using a combination of H and S channels of HSV color space and positive and negative Sobel edges.
The white lines are identified using a combination of V and S channels of HSC color space and positive and negative Sobel edges.
Sample unwarped lane line pixels image
After the binary lane line image is obtained, perspective transform is performed to turn the first person view image into a bird’s eyes view image, which will crop areas outside the region of interest as well.
The code for perspective transform is in
lane_detection.py. It computes source and destination points from input image as follows.
|Source x,y||Destination x,y|
|220, 698||320, 720|
|595, 454||320, 0|
|685, 454||960, 0|
|1060, 698||960, 720|
To verified that the perspective transform work as expected,
dst points are drawn onto a test image and its warped counterpart to verify that the lines appear parallel in the warped image.
Sample unwarped image with source points drawn
Sample warped image with dest points drawn
The sliding windows method is used to find where the mean of pixels are within the windows. Once found, a faster method without the sliding windows can be used.
The pixels are fitted to the polynomial equation Ay^2 +By + C.
The curvature radius and center of individual lane lines are calculated in
line_fit.py. It implements the formula (1 + (2Ay+B)^2 )^1.5 / abs(2A).
Sample pipeline output image with lane lines identified
python lib/calibrate.py # calibrate camera from images and save calibration matrix and distortion coefficients to `camera_cal.json` python test_camera.py # write images in output_images from running `undistort` from `camera.py` to calibration images python run.py # produces images in output_images from test_images folder python run.py video_in.mp4 video-out.mp4 # produces project and challenge video outputs
The approach to this project is much more elaborate than the simple algorithm used in project 1. So much time has been spent in this project to get the challenge video to work alright that I am actually too embarrassed to mention it. Much of it had to do with manual fine tuning of parameters and debugging. Had I knew of the time involved, I may have chosen to attempt this project with a deep learning approach. Reason being I do not believe this manual approach can get us to a level 5 autonomy. Thus although I gained knowledge in this project, I feel like the time would be better spent on a deep learning approach.
Next up is last project in course, project 5 - Vehicle Detection.