Skip to content

Research at St Andrews

Aggregating crowdsourced quantitative claims: additive and multiplicative models

Research output: Contribution to journalArticlepeer-review

Author(s)

Robin Wentao Ouyang, Lance M. Kaplan, Alice Toniolo, Mani Srivastava, Timothy J. Norman

School/Research organisations

Abstract

Truth discovery is an important technique for enabling reliable crowdsourcing applications. It aims to automatically discover the truths from possibly conflicting crowdsourced claims. Most existing truth discovery approaches focus on categorical applications, such as image classification. They use the accuracy, i.e., rate of exactly correct claims, to capture the reliability of participants. As a consequence, they are not effective for truth discovery in quantitative applications, such as percentage annotation and object counting, where similarity rather than exact matching between crowdsourced claims and latent truths should be considered. In this paper, we propose two unsupervised Quantitative Truth Finders (QTFs) for truth discovery in quantitative crowdsourcing applications. One QTF explores an additive model and the other explores a multiplicative model to capture different relationships between crowdsourced claims and latent truths in different classes of quantitative tasks. These QTFs naturally incorporate the similarity between variables. Moreover, they use the bias and the confidence instead of the accuracy to capture participants' abilities in quantity estimation. These QTFs are thus capable of accurately discovering quantitative truths in particular domains. Through extensive experiments, we demonstrate that these QTFs outperform other state-of-the-art approaches for truth discovery in quantitative crowdsourcing applications and they are also quite efficient.

Close

Details

Original languageEnglish
Pages (from-to)1621-1634
Number of pages14
JournalIEEE Transactions on Knowledge and Data Engineering
Volume28
Issue number7
Early online date29 Feb 2016
DOIs
Publication statusPublished - 1 Jul 2016

    Research areas

  • Crowdsourcing, Truth discovery, Quantitative task, Probabilistic graphical models

Discover related content
Find related publications, people, projects and more using interactive charts.

View graph of relations

Related by author

  1. Argumentation-based explanations of multimorbidity treatment plans

    Shaheen, Q., Toniolo, A. & Kuster Filipe Bowles, J., 2021, PRIMA 2020: Principles and Practice of Multi-Agent Systems: 23rd International Conference, Nagoya, Japan, November 18–20, 2020, Proceedings. Uchiya, T., Bai, Q. & Maestre, I. M. (eds.). Cham: Springer, p. 394-402 9 p. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); vol. 12568 LNCS).

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

  2. Responsible agent deliberation

    Walton, D. & Toniolo, A., 1 Jun 2020, Reason to Dissent: Proceedings of the 3rd European Conference on Argumentation, Volume . Novaes, C. D., Jansen, H., van Laar, J. A. & Verheij, B. (eds.). College Publications, p. 391-405 15 p.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

  3. Dialogue games for explaining medication choices

    Shaheen, Q., Toniolo, A. & Kuster Filipe Bowles, J., 2020, Rules and Reasoning: 4th International Joint Conference, RuleML+RR 2020, Oslo, Norway, June 29–July 1, 2020, Proceedings. Gutiérrez Basulto, V., Kliegr, T., Soylu, A., Giese, M. & Roman, D. (eds.). Cham: Springer, p. 97-111 15 p. (Lecture Notes in Computer Science (Programming and Software Engineering); vol. 12173 LNCS).

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

  4. On natural language generation of formal argumentation

    Cerutti, F., Toniolo, A. & Norman, T. J., 27 Dec 2019, Proceedings of the 3rd Workshop on Advances In Argumentation In Artificial Intelligence co-located with the 18th International Conference of the Italian Association for Artificial Intelligence (AI*IA 2019): Rende, Italy, November 19-22, 2019. Santini, F. & Toniolo, A. (eds.). Sun SITE Central Europe, p. 15-29 15 p. (CEUR Workshop Proceedings; vol. 2528).

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

  5. Dialectical models of deliberation, problem solving and decision making

    Walton, D., Toniolo, A. & Norman, T. J., 13 Sep 2019, In: Argumentation. First Online

    Research output: Contribution to journalArticlepeer-review

Related by journal

  1. IEEE Transactions on Knowledge and Data Engineering (Journal)

    William Vlcek (Reviewer)

    2014

    Activity: Publication peer-review and editorial work typesPeer review of manuscripts

Related by journal

  1. Towards real-time, country-level location classification of worldwide tweets

    Zubiaga, A., Voss, A., Procter, R., Liakata, M., Wang, B. & Tsakalidis, A., Sep 2017, In: IEEE Transactions on Knowledge and Data Engineering. 29, 9, p. 2053-2066 14 p.

    Research output: Contribution to journalArticlepeer-review

  2. Truth discovery in crowdsourced detection of spatial events

    Ouyang, R. W., Srivastava, M., Toniolo, A. & Norman, T. J., 1 Apr 2016, In: IEEE Transactions on Knowledge and Data Engineering. 28, 4, p. 1047-1060 14 p.

    Research output: Contribution to journalArticlepeer-review

  3. Development of a Software Engineering Ontology for Multisite Software Development

    Wongthongtham, P., Chang, E., Dillon, T. & Sommerville, I., Aug 2009, In: IEEE Transactions on Knowledge and Data Engineering. 21, 8, p. 1205-1217 13 p.

    Research output: Contribution to journalArticlepeer-review

ID: 247864425

Top