Below you can see the list of project ideas aligned with Harmony’s standards, all designed to elevate the functionality and accessibility of Harmony. These proposals aim to fortify Harmony as a comprehensive tool for researchers navigating questionnaire item harmonization across diverse studies. Each idea maintains a clear scope, typically avoiding extensive overhauls. Quick-start guidelines and beginner-friendly tasks are provided for each idea. We encourage contributors to join mentor teams and GSoC applicants to submit proposals aligned with these concepts.
We have a need to provide p-values instead of just the current Cosine similarity metric, in order to provide researchers with enhanced statistical insights, facilitating a more comprehensive analysis of harmonized questionnaire items
More information: https://github.com/harmonydata/harmony/issues
Skills: Statistical analysis, data interpretation, software development
Possible mentors: Thomas Wood
Expected size of the project: 175 hours
Difficulty: medium
Extend Harmony’s capabilities by integrating with advanced language models like GPT-3 and GPT-4, enabling more sophisticated natural language processing for improved harmonization
More information: https://github.com/harmonydata/harmony/issues
Skills: Natural Language Processing (NLP), API integration, and software development
Possible mentors: Thomas Wood
Expected size of the project: 175 hours
Difficulty: hard
Enhance Harmony to process negation in languages other than English and Portuguese, expanding its applicability and utility for researchers working with diverse language datasets
More information: https://github.com/harmonydata/harmony/issues
Skills: Multilingual NLP, language processing, software development
Possible mentors: Thomas Wood
Expected size of the project: 90 hours
Difficulty: medium
Address and fix issues related to processing PDFs within Harmony, as identified in the Kaggle competition. Improve PDF handling for more seamless integration
More information: https://github.com/harmonydata/harmony/issues/11
Skills: PDF processing, problem-solving, and software development
Possible mentors: Thomas Wood
Expected size of the project: 350 hours
Difficulty: hard
Integrate Harmony with prominent repositories like the UK Data Archive, UKLLC, and others. Enable seamless access and harmonization of questionnaire items from these repositories
More information: https://github.com/harmonydata/harmony/issues
Skills: Data repository integration, software development
Possible mentors: John Rogers
Expected size of the project: 175 hours per integration
Difficulty: medium
This project proposal aims to enhance the functionality of the Harmony project by implementing a new endpoint called “search” in Harmony’s API. This endpoint will enable third-party users, such as the UK Longitudinal Life Course Studies Consortium (UKLLC), to search for studies based on specific criteria. By integrating advanced search capabilities into Harmony’s API, users will be able to query the database and retrieve relevant studies efficiently.
search to Harmony’s API which would allow third parties such as UKLLC to search for studies. This is basically a URL that would probably look something like https://api.harmonydata.ac.uk/search and which would allow Harmony to receive a text and find variables in UKLLC’s database that are similar to it. The existing endpoints are here.Goals
Endpoint Development: Design and develop the “search” endpoint for Harmony’s API, allowing users to search for studies based on keywords, variables, or other search parameters.
Integration with UKLLC Database: Establish integration with the UKLLC database to enable seamless search functionality, ensuring compatibility and interoperability between Harmony and UKLLC data repositories.
Search Algorithm: Implement a robust search algorithm that efficiently identifies and retrieves studies matching the user’s search criteria, considering factors such as relevance, similarity, and accuracy.
Query Processing: Develop mechanisms for processing user queries, parsing search parameters, and executing database queries to retrieve matching studies from the Harmonybase.
Response Formatting: Define data structures and formats for organizing and presenting search results in a user-friendly manner, including study metadata, variables, and relevant information.
API Documentation: Create comprehensive documentation for the new “search” endpoint, including usage instructions, query syntax, and examples, to facilitate seamless integration and usage by third-party developers.
Expected Deliverables
Implemented “search” endpoint in Harmony’s API, accessible via the specified URL.
Integration with the UKLLC database to enable search functionality across studies.
Efficient search algorithm capable of retrieving relevant studies based on user queries.
Well-formatted search response containing detailed study information and metadata.
Comprehensive API documentation outlining endpoint usage and functionality.
More information: https://api.harmonydata.ac.uk/docs
Skills: Proficiency in web development, API design, and RESTful architecture. Familiarity with search algorithms, natural language processing, and information retrieval techniques
Possible mentors: Thomas Wood, Piyush Raj, John Rogers
Expected size of the project: 350 hours
Difficulty: medium
Harmony should also show an H-score, a similarity metric between two questionnaires.
More information: https://github.com/harmonydata/harmony/issues
Skills: Statistical analysis, software development
Possible mentors: Eve Cheng
Expected size of the project: 350 hours
Difficulty: medium
Allow Harmony to process raw survey data so that social scientists have everything they need to run the statistical analyses
More information: https://github.com/harmonydata/harmony/issues
Skills: Statistical analysis, software development
Possible mentors: Eoin McElroy
Expected size of the project: 350 hours
Difficulty: hard
Develop a feedback loop where users can flag bad matches and these are fed back into the matching algorithm
More information: https://github.com/harmonydata/harmony/issues
Skills: Statistical analysis, software development
Possible mentors: Thomas Wood
Expected size of the project: 350 hours
Difficulty: hard
Train a psychology-specific model where Harmony predicts the similarity between items in the way that a factor model would quantify the similarity
More information: https://github.com/harmonydata/harmony/issues
Skills: Statistical analysis, software development
Possible mentors: Eoin McElroy
Expected size of the project: 350 hours
Difficulty: hard
This project aims to leverage the Harmony project, a tool designed for retrospective harmonization of questionnaire items, to develop and manage Kaggle competitions focused on questionnaire item harmonization. Design Kaggle competitions centered around questionnaire item harmonization challenges, considering various aspects such as comparing items across studies, identifying the best match for item sets, assessing the compatibility of different questionnaire versions, and comparing items written in different languages.
Goals
Dataset Preparation: Prepare datasets for the competitions by curating diverse sets of questionnaire items from different studies, ensuring variability in item formats, languages, and versions.
Task Formulation: Define clear tasks and objectives for each competition, including specific harmonization goals, evaluation metrics, and guidelines for participants.
Competition Management: Oversee the entire competition lifecycle, including registration, submission, evaluation, and leaderboard management. Ensure fairness, transparency, and integrity throughout the competition.
Community Engagement: Foster a supportive and collaborative community of participants, mentors, and stakeholders interested in questionnaire item harmonization. Provide guidance, resources, and support to participants throughout the competition.
Evaluation Framework: Develop robust evaluation frameworks and scoring mechanisms to assess the effectiveness and quality of harmonization solutions submitted by participants.
Expected Deliverables
Well-designed Kaggle competitions addressing various aspects of questionnaire item harmonization.
Curated datasets representing diverse questionnaire items for use in competitions.
Clear documentation outlining competition objectives, guidelines, and evaluation criteria.
Managed competition platforms with active participation and engagement from the community.
Evaluation framework and scoring metrics for assessing participant submissions.
More information: https://harmonydata.ac.uk/harmony-on-kaggle/
Skills: Proficiency in data science, machine learning, and natural language processing. Experience with Kaggle competition design, dataset preparation, and competition management. Excellent communication and collaboration skills to engage with participants and stakeholders effectively.
Possible mentors: Thomas Wood from the Harmony project and Kaggle competition organizers will guide and support the selected GSoC participants throughout the project duration.
Expected size of the project: 350 hours
Difficulty: hard
This project focuses on integrating Pinecone (subject to change), an off-the-shelf similarity search service, into the Harmony project to facilitate efficient variable lookup functionality. Harmony aims to provide a lookup feature by indexing all variables from third-party sources offline and updating them periodically to ensure accuracy and relevance. Pinecone’s advanced search capabilities will enhance Harmony’s ability to quickly retrieve similar variables, enabling seamless data harmonization across studies.
In order to provide a lookup functionality, Harmony would need to index all the variables from a third party offline and do this every week or month. This means, converting them all to a vector and storing them somewhere so that we can quickly do a lookup. We might need an off the shelf tool such as Pinecone which would help us manage this lookup. Harmony is already doing this for the Mental Health Catalogue but the MHC case is simpler because there are fewer variables and we don’t have a process for constantly updating the database.
Goals
Pinecone Integration: Integrate Pinecone into Harmony’s infrastructure to support efficient variable lookup and similarity search functionalities.
Offline Indexing: Develop a process for offline indexing of variables from third-party sources, converting them into vectors, and storing them for rapid lookup.
Periodic Updates: Implement a mechanism for regularly updating the indexed variables to reflect changes and additions in the underlying data sources.
Lookup Functionality: Design and implement a lookup feature in Harmony’s user interface, allowing users to search for variables based on similarity and relevance.
Performance Optimization: Optimize the indexing and search processes to ensure fast and accurate variable lookup, even for large datasets and complex variable structures.
Expected Deliverables:
Pinecone integrated into Harmony’s backend infrastructure for efficient variable lookup.
Automated pipeline for offline indexing of variables from third-party sources and periodic updates.
User-friendly lookup interface in Harmony, enabling users to search for variables based on similarity.
Performance-optimized indexing and search algorithms for fast and accurate variable retrieval.
Skills: Proficiency in backend development, API integration, and experience with similarity search algorithms and vectorization techniques. Ability to work with off-the-shelf tools and services like Pinecone and integrate them into existing projects.
Mentors: Thomas Wood
Expected size of the project: 350 hours
Difficulty: hard
TODO: write description
These are proposals in the mailing lists and Discord which have not been published as project ideas yet. The feasibility is yet to be defined, and the idea may be dismissed depending on the feedback. Everyone is welcome to participate in the discussion and to join as a potential mentor.
List prepared in collaboration with Piyush Raj.
Sending data from another website to Harmony using Javascript We have exposed functionality for external websites to integrate with Harmony and add an “import to Harmony” button, either generated in Javascript or in Python. Create an Instrument object with at least an instrument_name and questions property in JSON - the questions must have a question_no and question_text properties eg: { "instrument_name": "Smoking behaviour", "questions": [ { "question_no": "1", "question_text": "Do you currently smoke or have you ever smoked?
Harmony at PyData London - 86th Meetup Update: you can download the slides from the presentation here Topic: NLP and generative models for psychology research Thomas Wood will present our work on Harmony, harmonydata.ac.uk, which is a free online tool that uses generative AI and LLMs to help psychologists analyse datasets. It uses Python, Pandas and HuggingFace Sentence Transformers to find similarities between questionnaires. Psychologists and social scientists often have to match items in different questionnaires, such as “I often feel anxious” and “Feeling nervous, anxious or afraid”.