A Conversational Agent
Powered by Crowdsourcing

Chorus is a Google Hangouts chatbot powered by crowdsourcing. A group of crowd workers will be collectively chat with you and try to solve your problems. You can now use Chorus on Google Hangouts for anything anytime, anywhere.

Inside Chorus, we created Evorus, a Crowd-AI framework that automates Chorus over time. Evorus won an Honourable Mention Award (top 5%) at CHI 2018.

Twitter

News

• Evorus got an Honourable Mention Award (top 5%) at CHI 2018 !
• Evorus is featured on WTAE-TV News: "Meet the 'Chorus' chatbot: Unlike Alexa or Siri, it's powered by actual people on the other end".
• Evorus is featured on CMU CS News: "Crowd Workers, AI Make Conversational Agents Smarter".
• Chorus is going to CHI ! Our paper on automating Chorus was accepted by CHI 2018. The pre-print is available now.

FAQ

1. Are real people responding to me?
   Mostly, yes, while Evorus currently automates 10% to 15% of conversations.
2. How do you recruit them? Do they just sit there and wait for us?
   We use Ignition model to recruit workers from Amazon Mechanical Turk (MTurk) to talk to you in real-time.
3. Who are the workers? Do they get paid?
   Yes, MTurk workers get paid to work. If you're interested in the demographics of MTurk workers, please take a look at mturk tracker.
4. How many workers am I talking to at the same time?
   It depends. We try to have up to 5 workers to talk to you at the same time, but sometimes we only manage to get one worker.
5. Why not just having one worker per conversation?
   We have multiple workers to vote on each other's responses to control the chat quality.
6. Would the workers know who I am?
   No. Workers only know that you are a "user" on Google Hangouts. (They can't even see your displayed name.)
7. Do I need to tell Chorus any of my personal information?
   No. You don't need to disclose any of your information to Chorus unless you want to.
8. What can I ask Chorus?
   We are exploring the possibilities of Chorus. So, you can be creative. You can say anything reasonable with Chorus. -- Just don’t abuse Chorus. Please keep in mind, they are actual humans.
9. Can I send images to Chorus?
   Yes. You can send images to Chorus via Google Hangouts, and the crowd workers will be able to see the image you sent.
10. Can I start a video call with Chorus?
    No. Chorus does not support video or audio calls.
11. Am I just talking to you?
    No, none of the researcher is involved in generating the responses.

Papers

• Machine Learning and Automation:
  Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time

  Ting-Hao K. Huang, Joseph Chee Chang, Jeffrey P. Bigham.

  In Proceedings of Conference on Human Factors in Computing Systems 2018 (CHI 2018), 2018, Montréal, Canada. (Acceptance Rate = 25.8%)

  Media Coverage: [CMU News] [EurekAlert!] [TribLIVE] [pressetext] [WTAE-TV News]

  Honourable Mention Award (top 5%) at CHI 2018

  [arXiv] [Video]
• Public Deployment:
  "Is there anything else I can help you with?": Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent

  Ting-Hao K. Huang, Walter S. Lasecki, Amos Azaria, Jeffrey P. Bigham.

  In Proceedings of Conference on Human Computation & Crowdsourcing (HCOMP 2016), 2016, Austin, TX, USA.

  Media Coverage: [The Register]

  [arXiv] [Slides]
• Original System Framework:
  Chorus: A Crowd-Powered Conversational Assistant

  W.S. Lasecki, R. Wesley, J. Nichols, A. Kulkarni, J.F. Allen, J.P. Bigham.

  In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2013). St Andrews, UK. p151-162.

  [Paper]
• Crowd-powered Entity Extraction for Conversational Agents:
  Real-time On-Demand Crowd-powered Entity Extraction

  Ting-Hao K. Huang, Yun-Nung Chen, Jeffrey P. Bigham.

  In Proceedings of the 5th Edition Of The Collective Intelligence Conference (CI 2017, oral presentation), 2017, New York University, NY, USA.

  [arXiv] [Data & Code] [Interface Demo] [Slides]
• "Guardian" is one of the follow-up works we developed to automate Chorus:
  Guardian: A Crowd-Powered Spoken Dialog System for Web APIs

  Ting-Hao K. Huang, Walter S Lasecki, Jeffrey P Bigham.

  In Proceedings of Conference on Human Computation & Crowdsourcing (HCOMP 2015), pages 62–71, November, 2015, San Diego, USA.

  [Paper] [Slides]

Team

Contact

Any questions, suggestions, proposals, opinions, or user feedbacks, please contact Ting-Hao (Kenneth) Huang at tinghaoh@cs.cmu.edu.

This field deployment study of the "Chorus" system is part of a research study conducted by Prof. Jeffrey P. Bigham and his group at Carnegie Mellon University.