RankTune
Overview
RankTune is a pseudo-stochastic data generation method for creating fairness-aware ranked lists using the fairness concept of statistical parity. Included in the RankTune module, it creates ranking(s) based on the phi representativeness parameter. When phi = 0 then the generated ranked list(s) does not represent groups fairly, and as phi increases groups are represented more and more fairly; thus phi = 1 groups are fairly represented. RankTune uses a pseudo-random process to generate fairness-aware ranked data. RankTune can generate ranked data from user provided group sizes, from existing datasets, along with producing relevance scores accompanying the ranked list(s).
Usage
RankTune can be utilized through four function interfaces.
Using Group Sizes
RankTune can be used to generate ranking(s) from group_proportions, a numpy array with each group's proportion of the total items,num_items, by using the GenFromGroups() function.
Usage:
import FairRankTune as frt
import numpy as np
import pandas as pd
from FairRankTune import RankTune, Metrics
#Generate a biased (phi = 0.1) ranking of 1000 items, with four groups of 100,
# 200, 300, and 400 items each.
group_proportions = np.asarray([.1, .2, .3, .4]) #Array of group proportions
num_items = 1000 #1000 items to be in the generated ranking
phi = 0.1
r_cnt = 1 #Generate 1 ranking
seed = 10 #For reproducibility
ranking_df, item_group_dict = frt.RankTune.GenFromGroups(group_proportions,
num_items, phi, r_cnt, seed)
#Calculate EXP with a MinMaxRatio
EXP_minmax, avg_exposures_minmax = frt.Metrics.EXP(ranking_df,
item_group_dict, 'MinMaxRatio')
print("EXP of generated ranking: ", EXP_minmax,
"avg_exposures: ", avg_exposures_minmax)
Output:
EXP of generated ranking: 0.511665941043515 avg_exposures: {0: 0.20498798214669187, 1: 0.13126425437156242, 2: 0.11461912123646827, 3: 0.10488536878769836}
Using an Existing Dataset
RankTune can be used to generate ranking(s) from item_group_dict, a dictionary of items where the keys are each item's group by using the GenFromItems() function.
Usage:
import FairRankTune as frt
import numpy as np
import pandas as pd
from FairRankTune import RankTune, Metrics
#Generate a biased (phi = 0.1) ranking
item_group_dict = dict(Joe= "M", David= "M", Bella= "W", Heidi= "W", Amy = "W",
Jill= "W", Jane= "W", Dave= "M", Nancy= "W", Nick= "M")
phi = 0.1
r_cnt = 1 #Generate 1 ranking
seed = 10 #For reproducibility
ranking_df, item_group_dict = frt.RankTune.GenFromItems(item_group_dict,
phi, r_cnt, seed)
#Calculate EXP with a MinMaxRatio
EXP_minmax, avg_exposures_minmax = frt.Metrics.EXP(ranking_df,
item_group_dict, 'MinMaxRatio')
print("Generated ranking: ", ranking_df)
print("EXP of generated ranking: ", EXP_minmax,
"avg_exposures: ", avg_exposures_minmax)
Output:
Generated ranking: 0
0 Nick
1 Dave
2 David
3 Joe
4 Nancy
5 Jane
6 Jill
7 Amy
8 Heidi
9 Bella
EXP of generated ranking: 0.5158099476966725 avg_exposures: {'M': 0.6404015779112127, 'W': 0.33032550440724917}
Generating Scores With the Ranking
Both GenFromGroups() and GenFromItems() contain sibling functions; respectively, ScoredGenFromGroups() and ScoredGenFromItems() that also generate relevance scores alongside the produced ranking(s). Scores are generated from either a normal or uniform distribution by setting the score_dist parameter to either normal or uniform.
For generating from group proportions use ScoredGenFromGroups() as follows:
import FairRankTune as frt
import numpy as np
import pandas as pd
from FairRankTune import RankTune, Metrics
#Generate a biased (phi = 0.1) ranking of 1000 items, with four groups of 100,
# 200, 300, and 400 items each.
group_proportions = np.asarray([.1, .2, .3, .4]) #Array of group proportions
num_items = 1000 #1000 items to be in the generated ranking
phi = 0.1
r_cnt = 1 #Generate 1 ranking
seed = 11 #For reproducibility
score_dist = "uniform"
ranking_df, item_group_dict, scores_df = frt.RankTune.ScoredGenFromGroups(group_proportions,
num_items, phi, r_cnt, score_dist, seed)
#Calculate EXP with a MinMaxRatio
EXP_minmax, avg_exposures_minmax = frt.Metrics.EXP(ranking_df,
item_group_dict, 'MinMaxRatio')
print("EXP of generated ranking: ", EXP_minmax,
"avg_exposures: ", avg_exposures_minmax)
Output:
EXP of generated ranking: 0.5218433930014378 avg_exposures: {0: 0.20212221456603452, 1: 0.13273063123258025, 2: 0.11380909457280128, 3: 0.10547614225010409}
For generating from existing items use ScoredGenFromItems() as follows:
import FairRankTune as frt
import numpy as np
import pandas as pd
from FairRankTune import RankTune, Metrics
#Generate a biased (phi = 0.1) ranking
item_group_dict = dict(Joe= "M", David= "M", Bella= "W", Heidi= "W", Amy = "W",
Jill= "W", Jane= "W", Dave= "M", Nancy= "W", Nick= "M")
r_cnt = 1 #Generate 1 ranking
seed = 10 #For reproducibility
phi = 0.1
score_dist = "uniform"
ranking_df, item_group_dict, score_df = frt.RankTune.ScoredGenFromItems(item_group_dict,
phi, r_cnt, score_dist,seed)
#Calculate EXP with a MinMaxRatio
EXP_minmax, avg_exposures_minmax = frt.Metrics.EXP(ranking_df,
item_group_dict, 'MinMaxRatio')
print("Generated ranking: ", ranking_df)
print("Generated scores: ", score_df)
print("EXP of generated ranking: ", EXP_minmax,
"avg_exposures: ", avg_exposures_minmax)
Output:
Generated ranking: 0
0 Nick
1 Dave
2 David
3 Joe
4 Nancy
5 Jane
6 Jill
7 Amy
8 Heidi
9 Bella
Generated scores: 0
0 0.771321
1 0.760531
2 0.748804
3 0.633648
4 0.498507
5 0.224797
6 0.198063
7 0.169111
8 0.088340
9 0.020752
EXP of generated ranking: 0.5158099476966725 avg_exposures: {'M': 0.6404015779112127, 'W': 0.33032550440724917}
How does it work?
RankTune is a fairness-tunable ranked data generation method. It constructs a ranking(s) ranking_df by placing items into the constructed ranking from top to bottom. The idea behind RankTune is that to construct a "fair" ranking, each time we place an item in the generated ranking, the likelihood of placing an item in a given group should be equal to that group's proportion of the total items (i.e., if a group is 20% of the item pool, then it should have a 20% chance of being placed). Then, on the other side of the spectrum, if we want a completely "unfair" ranking, we should place items into the rankings such that groups are ordered by increasing size from small to large. In this way, smaller groups get bigger proportions of favorable positions, which violates statistical parity fairness.
To generate rankings along the statistical parity fairness spectrum, RankTune samples a random number in the [0, 1] interval each time it places an item. We design this interval to have "regions" that map to groups. In this way, the unfairness tuning parameter phi controls representativeness, i.e., how fairly each group is represented in the ranking. Specifically, when phi = 1 , then each group is fairly represented. Thus each group's region is equal to the group's proportion of the pool (fair). As phi decreases, the fair representation of each group degrades because regions are distorted in such a way that smaller groups have larger regions compared to their proportion of the total pool (unfair). The fairness tuning parameter phi is used to create the regions prior to placing any items into the generated ranking.
RankTune Demonstration
The figure below displays the results of generating 200 rankings for different phi representativeness values for multiple group distributions. Average metric values (with \(95\%\) confidence intervals) are reported for 200 generated rankings per group distribution of \(1,000\) items. As phi increases, RankTune outputs increasingly fairer rankings. EXP and AWRF are measured with MinMaxRatio combo variables and are more fair at \(1\) (thus upward slopes), and NDKL, EE-D (EXP with an LTwo combo variable ), and ARP (with an MaxMinDiff combo variable) are more fair at \(0\) (thus downward slopes). To learn more, including the group distributions used, please see our technical report.
