Code It!

Develop, Automate, and Validate

Create New Account | Forgot Password?

This app created by the IDEA Lab

Overview

Code It! streamlines automated qualitative coding by combining keyword-based classifiers with statistical validation. Keyword-based classifiers allow for fair and transparent automated coding processes. Build, train, and validate your coding system with confidence using perfect sampling methodology.

Workflow

  1. Upload Data – Import your CSV or Excel file and select the column with data
  2. Create Code – Validate one code at a time. Define your code with a name, definition, and examples.
  3. Create Classifiers – Add keywords or regex patterns to identify your code.
  4. Training – Review examples and refine your classifier. Keep track of Cohen's Kappa, False Discovery Rate, and False Omission Rate.
  5. Validation – Achieve κ ≥ 0.80 through perfect sampling cycles.
  6. Code Dataset – Apply your validated classifier to all data and download final metrics.

Perfect Sampling Validation

The app uses a cycle-based perfect validation approach (Shaffer & Cai's 2024)

  • Calculates required sample size (Cai's N) based on your classifier's performance
  • Tracks consecutive perfect agreements between you and the classifier
  • Any disagreement ends the cycle, moves the item to training, and prompts classifier refinement
  • Validation is complete when you achieve the required number of consecutive agreements (κ > 0.80, α = 0.025)

This ensures statistical confidence before coding your full dataset.

Acknowledgments and References

Inspired by the Epistemic Analytics Lab and developed with assistance from Claude's Sonnet v4.5 LLM model.

Shaffer, D.W. & Cai, Z. (2024). Perfect Sampling.

Shaffer, D.W. & Ruis, A.R. (2021). How We Code.

Eagan, B. & colleagues. (2015). Can We Rely on IRR?

Arastoopour Irgens, G. & Eagan, B. (2023). The Foundations and Fundamentals of Quantitative Ethnography

Author

Golnaz Arastoopour Irgens

Recommended Citation

If you use Code It! in your research, please cite:

Arastoopour Irgens, G., Cai, Z., Eagan, B., Marquart, C., Ruis, A.R., Tan, Y., & Williamson Shaffer, D. (2025). Code It!: A web-based application for developing and validating automated qualitative coding systems. [URL]

Secure user authentication via AWS Cognito • Auto-saves your progress • Private user data storage

Upload Data

Next Step

Once you've uploaded your data and selected the text column, proceed to define your code.

Create Your Code

Define a single code for your analysis

Next Step

After defining your code name, definition, and examples, proceed to add classifiers/keywords.

Add Classifier for:


Classifier List

Keyword Suggester

After training your classifier, use Naive Bayes AI to suggest relevant keywords based on your coded examples. Requires at least 10 trained examples.

Suggested Keywords

These keywords appear more frequently in your positive examples. Click 'Add' to include them in your classifier.

Next Step

After adding your keywords/classifiers, proceed to train your classifier.

Training for:

Confusion Matrix

This shows how well the automated classifier matches your coding decisions

Training Metrics

These metrics tell you how well the automated classifier is performing

Train the Autocoder


Next Steps





Download Training Results Download Training Metrics
Ready for Validation?

Once you are satified with your Cohen's Kappa, FDR, and FOR, move on to validation.

Perfect Sampling

Validation complete when Kappa ≥ 0.80.

Any disagreement will immediately end this cycle and require classifier refinement.

Validation for:

Current Cycle Metrics

Overall Validation Metrics

Validation Item


? Validation Complete!

Kappa threshold reached. Your classifier is performing well!

Next Steps

Complete perfect sampling validation to proceed.

? Validation Complete!

κ > 0.80 achieved through perfect sampling

This will apply your validated classifier to all data and provide download.

Download Validation Results Download Validation Metrics