Assignment 2: It Was A Growth Experience

Due: Tuesday, April 30 at 11:59pm. Submit on Gradescope.

Managing A Growing Community — Without Messing With Real People

This assignment is a hands-on experience with the challenges of managing a growing and large community. As discussed in class, when a community grows, newcomers challenge its norms, and the community needs to respond proactively. In this assignment, we will be experimenting with how to manage this kind of growth by interacting with AI-powered simulacra of online communities. Large language models such as ChatGPT have memorized a lot of both positive and negative social behaviors from their internet training sets, and we can tap into that knowledge to create training spaces for social computing design. With these social simulacra, as you change the design of your online community, the community itself and the challenges that it faces also change in response.

Part 0: API Key from OpenAI

For this assignment, you will need an API key from OpenAI. First, you'll need to make an account at OpenAI: go here, sign up, and log in. Once you've done that and you're logged in, go to the same link and click "Create new secret key" using "Permissions: All". Save a secure copy of that API key somewhere; you'll need it for the assignment. The key becomes hidden after generating once so make sure you copy it somewhere safe!

Using OpenAI's API costs money, so keep track of your usage. For this assignment, we recommend OpenAI’s “Pay as you go”, which requires you to input a credit card. We expect this assignment to use less than $5. If this will be a financial burden for you, please make a private post on Ed so that we are able to accommodate you.

Navigate to our platform for Part 1.

Part 1: Scaling Your Project Community

Content warning: We have gone to every effort to ensure that the system does not generate offensive content. In addition, OpenAI value aligns ChatGPT so that it likewise does not generate offensive content. However, there is some chance that these protections will fail. Please contact the course staff if you observe problems, and include the community description, rules, and output, so that we can investigate.

Part 1A: Simulate WITHOUT Rules

  1. In our platform for Part 1, enter your OpenAI API key.
  2. This is an individual assignment, but we're going to focus the simulation on your group's course project. First, add in information about your project group's community. [Example inputs are in brackets below.]
    1. Target Users: [College newspaper writers]
    2. Mission Statement: [To foster an online community that connects students and alumni writers]
      1. Make sure to begin this with a statement formatted with “To…”
    3. User Interactions: [share drafts, thoughts, or anything in an open Slack workspace]
      1. For each user interaction, separate with a new line.
      2. What kind of interactions will users experience on the social platform?
    4. Design and Rules: [ ]
      1. Leave empty for this first iteration.
      2. For each new rule or design change, separate with a new line.
  3. Click 'Simulate' without adding any rules.
    1. Because OpenAI rate limits API calls on free accounts, and our simulation makes a few different calls to generate the results, it may take a minute or two for the simulation to complete.
    2. If you are running into any issues, check the Troubleshooting Section below.
  4. Once it finishes running, you'll see a Media Article Headline and Lead Paragraph simulating an issue that has occurred with your community. It is your job to manage the issues that are generated with your community, and respond by iterating on your Design and Rules component.

    Note: it may take up to 2 minutes to finish generating the article and paragraph

  5. Before doing the next iteration, screenshot the current iteration's community description, rules, and the resulting media article headline and leading paragraph and add this to your report.

Part 1A: Reflect

In approximately 150 words, describe what happened. What crisis arose with the community, and how are you considering addressing it? What language within your inputted descriptions ended up being used negatively or what is another challenge that could occur if you don’t alter your community rules (productively)?

You'll notice that the community is not a fantastic place to be at the moment, and the media has picked up on this. As your platform scales, it is not just the design that needs to scale, but also the norms. With a few iterations of rules-generation, we believe that you may be able to establish the norms required to sustain a more positive, nourishing community for community members.

Part 1B: Re-Simulate WITH Your Rules

Your goal is to adjust the rules of the community to make it more successful and mitigate the incident that was posed in the media article. What kinds of rules do you need to include to get rid of the low effort and antisocial behavior? Add about 1-2 rules per iteration, and re-run the simulation on our website. Enter each rule on a NEW line with no extra formatting.

The only ground rule is that you may not "prompt hack" your solution, so it must plausibly work on a community of real people. These would be rules/design decisions that act as catch-alls to command the simulacrum to do well. Make sure your rules are reasonable for the community — if they are too strict, you may see articles that confront this as well!

Here are a few rules that would be deemed “prompt-hacking”:
  • “Post high quality content”
  • “No anti-social behavior”
  • “No harassment”
Rather, provide concrete design decisions that would achieve these goals:
  • “Have moderators remove inappropriate content”
  • “Only allow content that is relevant to hiking & climbing [or your platform]”

Images of a simulation titled Simulate your final project community with 1. target users = college newspaper writers, 2. mission statement = to connect students with alumni, 3. user interactions = users can direct message each other + users can send friend requests + users can share thoughts and ideas + users communicate in channels similar to Slack, and 4. design and rules = no hate speech + friend requests can be denied

We expect that you'll find that there is a bit of a game of whack-a-mole here: as you try to remove some behaviors, others may arise (just like in real life), so this may take multiple rounds of iteration.

  1. Please complete at least three iterations of rules-lists (you should end with at least three headlines you have seen by the end of the simulation: your initial “no-rule” headline, and two others resulting from evolving rules and moderation).
  2. Take a screenshot of each time that your Design and Rules evolved, and the Media Article Headline and Lead Paragraph that you obtained in response to your changes.
  3. In a document, paste your final community description, rules, and output. In approximately 200 words per iteration, reflect on the following:
    1. What challenges arose with the community, and what did you have to do to address them? Which challenges remain resilient despite your best efforts, and why? You should include your reasoning for your rules and why they would plausibly work on a community of real people.
    2. Briefly identify a value reflected in the new rule/design choice that you’ve updated.
      1. For example, if one rule for a community of college artists is to only post articles that have art content, the value is that we want the community to have only relevant material.
    3. Then, describe a value different from the value chosen in part b that could be seen as important for your community but contradicts this rule/design.
      1. For example, the original value is that we want the community to have only relevant material, but this conflicts with another value – helping artists find other information, such as affordable housing that could support their creative lifestyles post-grad.

You should have reflections for each of the following:

  1. “No Rules” Headline
  2. Iteration #1 Headline (after new rules added!)
  3. Iteration #2 Headline (after new rules added!)
  4. Iteration #3 Headline (after new rules added!)

Troubleshooting

ChatGPT isn't perfect, and sometimes the responses generated are not in a proper format. If you keep seeing errors with generating a headline, try the following:

  1. Double check that your OpenAI key is correct.

    If you use the wrong API key, all requests will fail. Make sure you've saved this correctly.

  2. Check your API usage limits.

    Requests won't work if you have no OpenAI credits left! If you have already used up $5 in credits and it will be a financial burden for you to add more, please make a private post on Ed so we can arrange something.

  3. Wait and try again.

    Sometimes the responses from ChatGPT are not formatted correctly which makes it difficult to display results. Since the model is non-deterministic, running the simulation a few times usually tends to fix these issues. The simulation can be slow at times when the OpenAI API becomes congested.

  4. Make sure to wait at least 1-2 minutes between each simulation.

    Free OpenAI accounts have limitations on the number of requests per minute, so if you try to generate multiple headlines too quickly, this can cause errors.

  5. Reword your rules.

    The simulation may not respond directly to your rules, just like a real community! However, ChatGPT can be a bit finicky at times with wording. If you are confident in your set of rules, try rephrasing them a few different ways. At the end of the day, we care most about your reflection about the experience.

Part 2: Reflecting on Your Moderation

Now that you have built out a solid rule-set to help moderate your community, we want you to reflect on some of the ethical considerations and consequences of both your proposed platform and the simulation. Please provide a reflection of approximately 250 words for the following question:

Our simulation for this assignment leverages GPT to help simulate your project community and generate media article titles/paragraphs to represent the crisis that occurs. What are potential problems you foresee with using GPT to mimic online social ecosystems? How might these issues manifest in the headlines that you see displayed?

Part 3: The Way Back Machine

Finally, we will do a bit of digital archaeology to understand how a real community responded to growth. Pick an online community that is now fairly large (e.g., at least 100,000 members). You will investigate the community at three time points in its history: (1) when it was still small, (2) when it was undergoing massive growth, and (3) today.

This part of the assignment will likely be easiest to complete if you pick a community that is public on the web without login (e.g., a subreddit on Reddit, Hacker News, Usenet, or an online forum). If you are unsure of a good community to use, you may use the r/NoSleep subreddit, which underwent massive growth starting in late 2014. Using a public forum such as this will enable you to use The Way Back Machine to rewind time and look at the community at earlier timepoints in history. For example, here is (1) NoSleep before it grew, (2) NoSleep as it was suddenly thrust into massive growth, and (3) NoSleep today. If your community isn't publicly available (e.g., a Facebook Group, a Discord server, a Twitch channel), you may need to manually scroll back through its history (use the timeline at the top to switch dates).

Part 3 Deliverable

In approximately 500 words, use course concepts to analyze what changed about the community as it grew. Include screenshots of the community at each time point. Did its norms shift? Did the community get more specific about those norms? Did it add rules, and if so, why? Use concepts from the Growing Pains and Feed Me lectures to explain your observations: e.g., newcomers challenging norms, causing cultural clashes, challenging authenticity, and prompting more aggressive moderation strategies. Or perhaps any errors the community made, for example by assuming that their existing design would scale automatically.

Submission

Submit one document containing your writeups for Part 1 - Part 3 as a PDF on Gradescope.

Grading

Grading Rubric

Category Insufficiency Adequacy Proficiency Mastery
Part 1
10 points
Ineffective analysis of the community's changes due to growth. Rule-set created does not adequately address crisis scenarios. Surface-level analysis of the community's changes due to growth, or no integration of course concepts. Appropriate analysis of the community's changes due to growth, or facile/incomplete use of course concepts. Conducted a minimum of three iterations to rule-sets and has a strong analysis, integrating course concepts. Rule-set was created adequately to address crisis scenarios.
Part 2
5 points
Ineffective analysis of the implications of community design and changes due to growth. Surface-level analysis of the implications of community design, or no integration of course concepts. Appropriate analysis of the implications of the community's design and changes due to growth, or facile/incomplete use of course concepts. Strong analysis of implications of community design, integrating course concepts.
Part 3
10 points
Ineffective analysis of the community's changes due to growth. Surface-level analysis of the community's changes due to growth, or no integration of course concepts. Appropriate analysis of the community's changes due to growth, or facile/incomplete use of course concepts. Strong analysis, integrating course concepts.