The goal of this project was to develop an automatic algorithm that would suggest crops of an input image so that the overall composition of the image improves. The target are amateurs who may not be familiar with composition rules, but also hobbyists and professionals who may have to use center focusing and then have to crop many photos later or who may be interested in various crops suggested by our algorithm as an inspiration.
We have had 4 interns working on the problem.
- With Armin Samii in the summer of 2012. There are many rules to creating a well composed photograph, and many more exceptions to each of these rules. Trying to encode all of these rules and exceptions is likely intractable. Instead, we proposed a data driven approach: by using a database of well composed exemplar images, we can crop a new image to match the composition of an exemplar. Assuming that the database implicitly encodes many rules and their exceptions, we eliminate the need for a rule based approach. We used an algorithm that encodes a neighborhood of points on object boundary and used that information to search in the database of images. The resulting algorithm was published in Computer Graphics Forum, 33(1), 2015.
- With Peng Wang in the summer of 2013. Instead of database search approach we tried with Armin we used machine learning to learn the crops from the database as well as the set of pairs of images and cropped images. We published the results in WACV 2015.
- Subsequently, we had an intern Chen Fang working in the summer 2013, who developed even more robust algorithm and we were using that one when showing the technology to product teams. Xiaohui Shen joined the team after he started in the summer 2013. That work was published in ACM Multimedia 2014.
- To productize the matlab code from summer 2014 we hired a 4-th intern, Jianming Zhang in Q1 2014 who not only converted the code to C++ but also developed some crucial improvements, making the algorithm quite robust. The resulting code shipped in Photoshop Elements.