Loans Applications? New techniques to measure social bias in software

Banks are increasingly using software to decide who will get a loan, courts to judge who should be denied bail, and hospitals to choose treatments for patients. These uses of software make it critical that the software does not discriminate against groups or individuals, say computer science researchers at the University of Massachusetts Amherst.

Professor Alexandra Meliou in the College of Information and Computer Sciences says, “The increased role of software and the potential impact it has on people’s lives makes software fairness a critical property. Data-driven software has the ability to shape human behavior: it affects the products we view and purchase, the news articles we read, the social interactions we engage in, and, ultimately, the opinions we form.”

Meliou with professor Yuriy Brun and Ph.D. student Sainyam Galhotra, have developed a new technique they call “Themis,” to automatically test software for discrimination. They hope Themis will empower stakeholders to better understand software behavior, judge when unwanted bias is present, and, ultimately improve the software.

Brun says, “Unchecked, biases in data and software run the risk of perpetuating biases in society. For example, prior work has demonstrated that racial bias exists in online advertising delivery systems, where online searches for traditionally-minority names were more likely to yield ads related to arrest records. Such software behavior can contribute to racial stereotypes and other grave societal consequences.”

The researchers’ paper describing this research, published in pre-conference materials for the European Software Engineering Conference (ESEC/FSE 2017) before its September meeting in Paderborn, Germany, has won an Association of Computing Machinery Special Interest Group on Software Engineering (ACM SIGSOFT) Distinguished Paper Award. The work is supported by the National Science Foundation.

Brun explains that while earlier research has considered discrimination in software, Themis focuses on measuring causality in discrimination. Software testing allows Themis to perform hypothesis testing, to ask such questions as whether changing a person’s race affects whether the software recommends giving that person a loan, he says.

“Our approach measures discrimination more accurately than prior work that focused on identifying differences in software output distributions, correlations or mutual information between inputs and outputs. Themis can identify bias in software whether that bias is intentional or unintentional, and can be applied to software that relies on machine learning, which can inject biases from data without the developers’ knowledge,” he adds.

When evaluated on public software systems from GitHub, Themis found that discrimination can sneak in even when the software is explicitly designed to be fair. State-of-the-art techniques for removing discrimination from algorithms fail in many situations, in part because prior definitions of discrimination failed to capture causality, the researchers point out.

For example, Themis found that a decision-tree-based machine learning approach specifically designed not to discriminate against gender was actually discriminating more than 11 percent of the time. That is, more than 11 percent of the individuals saw the software output affected just by altering their gender.

Themis also found that designing the software to avoid discrimination against one attribute may increase discrimination against others. For example, the same decision-tree-based software trained not to discriminate on gender discriminated against race 38 percent of the time

When robots help with shopping

The IFL PiRo robot detects the objects desired and automatically puts them into shopping baskets.
Credit: Laila Tkotz, KIT

Today, the desired book, toy or household appliance can be purchased by a click only — thanks to online mail order business and smart logistics. The bottleneck in logistics, however, is the high-bay store, where many picking and detection processes cannot yet be executed automatically by robots. At the Amazon Robotics Challenge in Nagoya, Japan, the IFL PiRo team of KIT demonstrated how future warehousing may work. First participation in the international competition was crowned by an excellent 7th place in the overall ranking.

“In an exciting week with three intensive competition days we mastered several challenges and learned a lot,” says Kai Markert of KIT’s Institute for Materials Handling and Logistics and IFL PiRo team captain. “It was a big success for our young team to be able to compete with the best in the world in Japan.”

The IFL PiRo team presented with an innovative shelf concept. Instead of arranging the goods in a conventional shelf, the objects are placed in mobile boxes arranged horizontally around the central robot arm. Some of the boxes can be moved by the robot like drawers, thus allowing for a multistorey arrangement. Although load capacity of this system appears to be smaller than that of a shelf system of the same area at first glance, the capacity can be increased easily. In the future warehouse, a second robot might work as a feeder and bring the required boxes or remove those that are no longer needed. “As today’s warehouse systems have reached their technical limits, we wanted to develop a completely new system for the 21st century in order to make full use of the advantages of the robot gripper,” Markert points out.

The system was designed such that all actions, i.e. detection, picking, suction, dropping, can be carried out with similar movement patterns from above. This facilitates movement planning and execution and makes the movements quicker and more reliable. Moreover, a laser scanner as used in driverless transportation systems can monitor the area above the boxes to control the success of picking, suction, and transport.

To pick the products, the robot uses a gripper or a suction pad. Depending on the product to be picked, the control independently decides on the method to be used and selects the right parameters and contact points. A camera system supplies two- and three-dimensional image information. Using image recognition software and neural networks, the objects are detected. Grip points can be given when teaching the system or calculated from the position and size of the object after object detection.

The IFL PiRo team consists of about 15 students and scientific staff members of the KIT Institute for Materials Handling and Logistics. The team members work in the areas of mechanical engineering, precision engineering/mechatronics, electrical engineering, computer science, and business engineering. Apart from the IFL PiRo system concept and the software, many mechanical parts were self-developed and produced at the own workshop or using 3D printers.

Software lets designers exploit the extremely high resolution of 3-D printers

Today’s 3-D printers have a resolution of 600 dots per inch, which means that they could pack a billion tiny cubes of different materials into a volume that measures just 1.67 cubic inches.

Such precise control of printed objects’ microstructure gives designers commensurate control of the objects’ physical properties — such as their density or strength, or the way they deform when subjected to stresses. But evaluating the physical effects of every possible combination of even just two materials, for an object consisting of tens of billions of cubes, would be prohibitively time consuming.

So researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a new design system that catalogues the physical properties of a huge number of tiny cube clusters. These clusters can then serve as building blocks for larger printable objects. The system thus takes advantage of physical measurements at the microscopic scale, while enabling computationally efficient evaluation of macroscopic designs.

“Conventionally, people design 3-D prints manually,” says Bo Zhu, a postdoc at CSAIL and first author on the paper. “But when you want to have some higher-level goal — for example, you want to design a chair with maximum stiffness or design some functional soft [robotic] gripper — then intuition or experience is maybe not enough. Topology optimization, which is the focus of our paper, incorporates the physics and simulation in the design loop. The problem for current topology optimization is that there is a gap between the hardware capabilities and the software. Our algorithm fills that gap.”

Zhu and his MIT colleagues presented their work this week at Siggraph, the premier graphics conference. Joining Zhu on the paper are Wojciech Matusik, an associate professor of electrical engineering and computer science; Mélina Skouras, a postdoc in Matusik’s group; and Desai Chen, a graduate student in electrical engineering and computer science.

Points in space

The MIT researchers begin by defining a space of physical properties, in which any given microstructure will assume a particular location. For instance, there are three standard measures of a material’s stiffness: One describes its deformation in the direction of an applied force, or how far it can be compressed or stretched; one describes its deformation in directions perpendicular to an applied force, or how much its sides bulge when it’s squeezed or contract when it’s stretched; and the third measures its response to shear, or a force that causes different layers of the material to shift relative to each other.

Those three measures define a three-dimensional space, and any particular combination of them defines a point in that space.

In the jargon of 3-D printing, the microscopic cubes from which an object is assembled are called voxels, for volumetric pixels; they’re the three-dimensional analogue of pixels in a digital image. The building blocks from which Zhu and his colleagues assemble larger printable objects are clusters of voxels.

In their experiments, the researchers considered clusters of three different sizes — 16, 32, and 64 voxels to a face. For a given set of printable materials, they randomly generate clusters that combine those materials in different ways: a square of material A at the cluster’s center, a border of vacant voxels around that square, material B at the corners, or the like. The clusters must be printable, however; it wouldn’t be possible to print a cluster that, say, included a cube of vacant voxels with a smaller cube of material floating at its center.

For each new cluster, the researchers evaluate its physical properties using physics simulations, which assign it a particular point in the space of properties.

Gradually, the researchers’ algorithm explores the entire space of properties, through both random generation of new clusters and the principled modification of clusters whose properties are known. The end result is a cloud of points that defines the space of printable clusters.

Establishing boundaries

The next step is to calculate a function called the level set, which describes the shape of the point cloud. This enables the researchers’ system to mathematically determine whether a cluster with a particular combination of properties is printable or not.

The final step is the optimization of the object to be printed, using software custom-developed by the researchers. That process will result in specifications of material properties for tens or even hundreds of thousands of printable clusters. The researchers’ database of evaluated clusters may not contain exact matches for any of those specifications, but it will contain clusters that are extremely good approximations.