Beautiful brushstrokes drawn from data

(Press-News.org) A good painter uses simple strokes of a brush to bring texture, contrast and depth to a blank canvas. In comparison, computer programs can have difficulty reproducing the complex and varied forms of brushstrokes, and often require painstaking effort to mimic a brief sweep of paint.

Now, a team of researchers including scientists at Princeton University has developed a program that allows graphic artists to quickly and easily produce realistic brushstrokes on their computers. Called RealBrush, the program combines graphics algorithms with "Big Data" storage and retrieval techniques to allow computer artists to create, bend and shape a wide array of brushstrokes. Besides creating the strokes, the program allows for effects including smudging, smearing and merging of different types of media.

"Our goal is to have it look like a photograph of a real stroke but to have it follow whatever path you happen to be drawing," said Adam Finkelstein, a computer science professor and the senior researcher on the project. "The challenge is if you have a photo of a paint stroke, how do you make it fit a different path?"

As a first step, a RealBrush user creates a few sample strokes using any desired medium, from oil paint to nail polish or even toothpaste. The user then photographs the strokes and enters them into RealBrush's library (the program also has a variety of pre-installed strokes in different media.) The program uses the sample brushstrokes as baselines indicating fundamental characteristics of the strokes — for instance, the program looks at the center line, or spine, of a segment to help determine a basic shape. The program can use those samples to warp and blend the original strokes into any curves or forms the user desires.

"We use feature matching — it is a machine-learning approach," said Jingwan "Cynthia" Lu, a graduate student in computer science who was the lead author of a paper the researchers presented at the SIGGRAPH computer graphics conference in Anaheim, Cal., in August. "Based on the shape of the spine, we want to find segments of the exemplar that have a similar spine and use the segments to make similar shapes."

In this video, Lu discusses her research and how it allows her to bring her interests in various art forms to her work.

RealBrush uses a comparable approach to simulate other graphic effects: "shape," the silhouette and texture of a single stroke; "smear," a new stroke applied across wet paint, which causes a mingling of colors; and "smudge," the effect of a dry brush or finger rubbed across a brushstroke. The researchers left one more feature for future work — "composite," which is the glazing or reflective effect of translucent paints layered on top of each other.

"For each style, we need about 20 to 30 strokes," Lu said. "If you are a casual user, you can use pre-captured strokes. Or you can paint your own strokes and record those in our system. You can share those with your friends."

Lu worked out the basics of RealBrush during an internship working with a senior researcher, Stephen DiVerdi, at Adobe Systems in San Francisco. As part of the project, she collaborated with Finkelstein; Connelly Barnes, a Princeton 2011 Ph.D. recipient now transitioning from a postdoc at Adobe to a faculty position in computer science at the University of Virginia; and DiVerdi, who is now at Google, to create the paper that was presented at SIGGRAPH. Although RealBrush is a working prototype, Finkelstein said it is a research project, and he noted that it would take additional effort before it could be available for widespread use.

Finkelstein said the RealBrush approach is an example of a change in programming that has the potential for huge impact in many fields, including graphic design.

"In the old days, people wrote clever algorithms to simulate what they were trying to achieve," he said. "What algorithm can I use that will look like a brush on canvas? It was hard, and it met with limited success."

Now, he said, graphics programming is "entering the world of Big Data."

Big Data — a term for the paradigm shift in which the ability to capture, store and analyze huge quantities of digital information allows scientists to leverage this data in addressing new challenges — is fundamentally changing many fields of computing.

A Big Data approach allows programmers to use that data to answer questions, or create simulations, through examples rather than pure mathematics. To do that, scientists need to use statistical and machine-learning approaches to draw the answers from the vast amounts of data available.

"How do you draw the fundamental characteristics of paint from many examples of paint? How do you extrapolate traffic patterns from what traffic was like today, or last week, or last month?" Finkelstein said. "We are entering a realm where you don't have to apply a clever algorithm to simulate the world; you can say, 'this is how the world really is.'"



INFORMATION: