Final Project
CS180: Intro to Computer Vision and Computational Photography

Pyramid-Based Texture Analysis/Texture:
Matching and generating texture using image pyramids

Background

This project revolves around re-implementing key portions of the paper "Pyramid-Based Texture Analysis/Texture". This paper uses a slightly modified laplacian pyramid, which will decompose the image into frequency bands AND orientations in the frequency domain (can imagine splitting the plotted frequency domain amplitudes into our circular frequency bands AND quadrants). By using histogram matching, we are able to generate oriented textures matching the source image. By matching histograms of not just the source and noisy images, but also the oriented laplace pyramid, we are able to generate a texture image which has actual structure to it instead of just pixels with similar values.

Overview

The goal of this project is to try and re-create the basic results of this paper in mimicing a 2D texture. You will lean on your previous code for the laplacian pyramid, making small modifications to get orientation bands. You will also implement the algorithm described with pseudocode in the paper. If everything goes right, you will be able to showcase your ability to match textures!

Details

For starters, you will need to use the oriented filters talked about in the paper. In many implementations and in the steerable filters paper, the filters are defined in the frequency domain. For simplicity, we will take the approach of defining the filters in the spatial domain. To get these filters, you can use pyrtools's (a python pyramid tools library) pre-calculated oriented filters, as shown here. While you can use the filters from this library, we still expect you to implement the steerable pyramid yourself, so these filters are the only part of the library you are allowed to use.

Once you have the aligned filters working, you will need to implement the oriented pyramid. This should be similar to the laplacian pyramid, and you can follow the paper's diagram for how to construct it. You will start with one laplace layer, and on the low frequency component, you will recurse with the oriented pyramid.

With the actual pyramid working, you will need to implement the histogram matching, which is named match-histogram in the paper and is outlined in pseudocode. Because this is all cumulative, make sure to unit test this code and all your code before. Make sure the histogram for some randomly initialized gaussian noise ends up matching the histogram of some source image of your choosing, and make sure before continuing that your oriented laplace pyramid is behaving as expected in the way it was demonstrated to work in the paper with the pac-man-like example (figure 4).

Finally, you will need to implement the texture synthesis algorithm, which is also outlined in pseudocode, named match-texture. This will require you to use all the code you've implemented up until now.

Some extra considerations are needed to make this work. You will likely need to use same or wrap padding when convolving, and color images need to be dealt with properly. As described in the paper, for color images you will need to transform the 3-dimensional pixel values (RGB) of your source image into a PCA basis in order to de-correlate and normalize the pixel values, enabling you to operate on the RGB channels independently. For extra reading into practical considerations and greater detail on pseudocode (ignore their math for the filters, the library we provided you should be all you need) and other misc, see this link here, but this shouldn't be required to get the basic code running.

Bells & Whistles (Extra Credit)

Here are some ideas, but we will give credit for other clever ideas:

• Hyperparameter tuning. We made several design choices: operating in the spatial domain instead of the frequency domain, the kind of padding we used, the number of iterations, the number of oriented filters, etc. Ablate some of these and compare their performance.
• Implement the 3D texture map generation that the paper shows off

For all extra credit, be sure to demonstrate on your web page cases where your extra credit has improved image quality.

Deliverables

• For every assignment you should create a main.py, main.ipynb, or main.m that can be used to run all your code for the assignment, and a README.txt file that contains a description of each code file, your name, SID, school email. Submit all source code used to generate your results, as well as the README.txt file, to Gradescope (Entry Code: 57Y3YJ). Please be sure to appropriately comment your code so that the graders can appropriately identify where your implementations lie and what they do.
• Additionally, upload a web page index.html as the report:
• Text giving a brief overview of the project, and text describing your approach. If you ran into problems on images, describe how you tried to solve them. The website does not need to be pretty; you just need to explain what you did.
• Explain the oriented filters, and show how the sum of all oriented filters for a given band is the same as a standard laplacian band pass. Show some sample images convolved with the oriented filters.
• A clear visual demonstration of the oriented pyramid. Take some sample image of your choosing and demonstrate that each of your orientations correspond to different oriented frequencies.
• A clear visual demonstration of the histogram matching. Take some sample image of your choosing and demonstrate that the histogram of the noisy image matches the histogram of the source image. This will likely still look like noise, but the colors should roughly match (not to be confused with texture which will not match). This can be greyscale or color, but is best demonstrated with color images.
• The result of your full match-texture algorithm on a few examples of your own choosing.
• If your algorithm failed on any images, show them as well, and talk about why you think they didnt work so well.
• Describe any bells and whistles you implemented. For maximum credit, show before and after images.
• Remember not to use any absolute links to images etc on your computer, as these will not work online. Only use relative links within your folder.

This assignment will be graded out of 100 points, as follows:

• 20 points for demonstrating how the oriented filters work when colvolved on an image
• 40 points for demonstrating the oriented pyramid.
• 20 points for demonstrating the histogram matching.
• 20 points for demonstrating the full texture synthesis.