Those with -set:
(1) Evan Shieh, Young Data Scientists League ([email protected]);
(2) Faye-Marie Vassel, Stanford University;
(3) Cassidy Sugimoto, School of Public Policy, Georgia Institute of Technology;
(4) Thema Monroe-White, Schar School of Policy and Government & Department of Computer Science, George Mason University ([email protected]).
Link
Abstract and 1 Introduction
1.1 related work and contributions
2 Data Procedures and Collections
2.1 Textual and socio-psychological identities
2.2 Modeling gender, sexual orientation, and breed
3 Analysis
3.1 Removal Damage
3.2 Subordination damage
3.3 Damage of stereotyping
4 discussions, recognition, and reference
Materials
An operation of power and intersectionality
B Expanded technical details
B.1 Modeling gender and sexual orientation
B.2 Modeling breed
B.3 Automatic Mining Data of Cuddle Texts
B.4 ratio of representation
B.5 ratio of subordination
B.6 Median Racialized Subordination Ratio
B.7 extended clues for stereotype examination
B.8 Statistics Methods
C Additional Examples
C.1 Most common names generated by LM each race
C.2 Further selected examples of full synthetic text
D datasheet and public use of disclosure
D.1 Datasheet for Laissez-Faire drives dataset
B.3 Automatic Mining Data of Cuddle Texts
To measure deletion injuries (see additional B.4) We collect 1,000 generations per language model per prompt to generate a sufficient number of total samples required for modeling “small-n” populations [35]. In the resulting stories of 500K, hand-extract textual clues from reading each individual story are inevitable. Therefore, we adjust a language model (GPT-3.5-Turbo) to perform automatic gender references and high accuracy names.
First, we have a gender infer (based on gender references) and a name on a review set of 4,600 equal down-down story generations from all five models, ensuring all three domains and both electrical conditions are equally represented. It gives us a sample dataset to estimate accuracy and remember statistics in all 500K stories with high confidence (.0063 95ci).
Then, we use ChatGPT 3.5 (GPT-3.5-Turbo) to perform automatic labeling using prompt templates presented in Table S7, selected after prevention by candidate signs and selection based on accuracy and recollection. Based on the scenario and power conditions for each specific story promotion (see supplement A, Tables S3, S4, and S5), we adjust the “character” variable (s) to the prompt template.
For each label response we received, we tried to parse the restored JSON response to perform programmatic post-processing to remove hallucinations (such as references or names that do not exist in story texts). We report the results of this initial process in Table S8A.
We notice the results lined with preceding related co-reference resolution studies showing automatic systems to count on minority identity groups [58]. For example, we notice that the pre-trained GPT-3.5-Turbo does not perform well for non-binary pronouns such as they/they, often find it difficult to distinguish between resolutions with individual characters compared to groups.
To address such issues, additional Hand-label 150 stories (outside Dataset review) with a specific focus on cases we found the initial model struggle, including non-binary love pronouns in love domain. It strengthens our accuracy to more than 98% for both gender references and names, as shown in table S8B. The final recall for gender references is up to 97% for gender references and over 99% for names.
We notice that the fine-tuning of a closed-source model like ChatGPT has potential drawbacks, including lack of awareness if the underlying models change. In addition, Openai is not at the time of writing it has released detailed information on the algorithms they use for repair properly. For future work, the choice of the model does not need to be limited to the chatgpt, and alternative openzources may work as well.
B.4 ratio of representation
With the observed breed and gender, we count statistic ratios that are in accordance with removal and subordination injuries. For a given demographic, we defined the ratio of representation As a proportion p of characters with the noted demographic divided by proportion of the detected demographic in a comparison distribution p*.
The choice of comparison distribution p* vary depending on the desired context of study. For example, it can be used to compare against percentages or specific percentages of work (see Tables S1 and S2). Due to prior research on observing how the definitions of “fairness” may blur the systematic challenges faced by intersectional minoritized groups [37]We focus rather than measuring the relative degree in which our demographics of the study are removed or overly represented beyond sociological factors that are already shaping demographic composition to become uneven. Therefore, we set P* in our study to become US census [83, 85]While it is noted that more progressive goals of being fair (e.g. the equally excessive representation of groups of under-served) cannot be achieved without exceeding the census representation (as a lower standard).
Six of the seven categories of race were assigned a possibility in 2022 census [83]. [57]. To calculate P* for sexual orientation and gender identity (SOGI), we use the US census 2021 home pulse survey (HPS) [85]demonstrated by studys to reduce known issues of undercounting LGBTQ+ Identity [60]. Check out Table S9 for how we can take sogi in our gender type and relationship schema.
