- Level Intermediate
- Course by Johns Hopkins University
-
Offered by
About
In the course "Training AI with Humans", you'll delve into the intersection of machine learning and human collaboration, exploring how to enhance AI performance through effective data annotation and crowdsourcing. You’ll gain a comprehensive understanding of machine learning principles and performance metrics while developing practical skills in using platforms like Amazon Mechanical Turk (AMT) for crowdsourced tasks. This unique approach combines theoretical knowledge with hands-on experience, allowing you to implement Inter-Annotator Agreement (IAA) techniques to ensure high-quality annotated data.
By completing this course, you will be well-equipped to design and conduct impactful crowdsourcing studies, improving AI models in real-world applications such as healthcare and research. Whether you're looking to enhance your skills in machine learning, optimize data collection processes, or understand the ethical implications of crowdsourcing, this course offers invaluable insights and tools.
Skills you learn
Modules
Untitled Lesson
1
Readings
- Course Overview
Introduction to Machine Learning
1
Assignment
- Introduction to Machine Learning
2
Videos
- Machine Learning
- Models
1
Readings
- Reading References
Evaluating and Constructing ML Classifiers
1
Assignment
- Evaluating and Constructing ML Classifiers
3
Videos
- Operationalize Data
- Data Normalization
- Decision Tree
1
Readings
- Reading References
Module-end Assessments
1
Assignment
- Machine Learning
1
Labs
- Practice Lab - Machine Learning Classifier to Predict in R
Understanding Inter-Annotator Agreement (IAA)
1
Assignment
- Understanding Inter-Annotator Agreement (IAA)
2
Videos
- Inter-Annotator Agreement (IAA) Examples
- Inter-Annotator Agreement (IAA) Measures
1
Readings
- Reading References
Calculating and Implementing IAA
1
Assignment
- Calculating and Implementing IAA
1
Videos
- Inter-Annotator Agreement (IAA) Calculation
1
Readings
- Reading References
Module-end Assessments
1
Assignment
- Inter-Annotator Agreement (IAA)
Introduction to Crowdsourcing
1
Assignment
- Introduction to Crowdsourcing
2
Videos
- Crowdsourcing
- Amazon Mechanical Turk
Setting Up and Designing Crowdsourcing Tasks
1
Assignment
- Setting Up and Designing Crowdsourcing Tasks
2
Videos
- Experimentation
- Tutorial on setting up your first AMT account
Module-end Assessments
1
Assignment
- Crowdsourcing
1
Labs
- Practice Lab: Impact of Payment & Complexity on Crowdsourcing Task Efficiency
1
Readings
- Reading References
Designing Crowdsourcing Studies with AMT
1
Assignment
- Designing Crowdsourcing Studies with AMT
1
Videos
- Design of Experiments
1
Readings
- Reading References
Collecting and Analyzing AMT Data
1
Assignment
- Collecting and Analyzing AMT Data
1
Videos
- AMT Addiction
1
Readings
- Reading References
Module-end Assessments
1
Assignment
- Platforms
1
Labs
- Practice Lab: Neuroscientific Explanation of Addiction - Analyzing Stigma, Dangerousness, & Social Distance
1
Readings
- Self-Reflective Reading: Personal Reflection on Platforms
Impact of Inter-Annotator Agreement on ML Performance
1
Assignment
- Impact of Inter-Annotator Agreement on ML Performance
2
Videos
- Data Myths and the R.O.A.D. Framework
- Case Study: COVID Test Kit Mailing
1
Readings
- Reading References
Designing Effective Crowdsourcing for ML Improvement
1
Assignment
- Designing Effective Crowdsourcing for ML Improvement
2
Videos
- Case Study: Organ Transplant
- Case Study: COVID Case Count Estimation
1
Readings
- Reading References
Module-end Assessments
1
Assignment
- Crowdsourcing and Machine Learning
1
Readings
- Self-Reflective Reading: Crowdsourcing and Machine Learning
Ian McCulloh