development

Harmony Data Discovery

Harmony Data Discovery

New Feature in Development: Enhancing Harmony with User-Centred Discovery

At Harmony, we’re working on a new feature aimed at making data discovery and exploration - through the use of meta-data - even more efficient and intuitive To ensure this feature addresses real needs, we’re conducting co-design sessions with researchers, data-managers and other users, allowing us to develop a tool that solves real-life user requirements.

The Co-Design Approach: Co-design allows us to build this feature with direct input from those who will use it most. Through these sessions, users and stakeholders participate in shaping the tool, providing detailed feedback, identifying challenges, and contributing their own insights. This process is collaborative by design, creating a feedback loop that helps us address pain points in data discovery and visualisation while ensuring usability.

What problem does Harmony Discovery solve? Researchers can struggle to find datasets and often need something specific, such as a longitudinal study measuring anxiety and household income. Existing repositories don’t always go into the details of which variables have been captured, and finding variables can be a difficult process with a basic keyword search. Harmony Discovery indexes studies, datasets, and variables using the same cutting-edge vector representation used in our harmonisation tool, so that a search for “anxiety” will return studies with the question items “I feel anxious”, “I often feel worried”, and even items in other languages. This is made possible with the latest developments in AI and large language models (LLMs), which allow us to index and retrieve items based on semantic content rather than just matching words.

Key Goals for the New Feature: While the feature is still evolving, our main objective is to understand how researchers search and discover research data and how Harmony can support the decision making process especially when trying to compare meta-data of large or complex datasets. From early feedback, we’ve noted specific needs for adaptability and efficiency in data discovery and comparison, and we’re integrating these priorities into the design of the tool.

Get Involved: If you’re a researcher or data manager interested in participating in our co-design workshops, we’d love to hear from you! Get in touch with us via email, you can also sign up to our Discord server and newsletter and follow us on LinkedIn to stay connected with the latest updates and community discussions.

Harmony is an open source tool for social science research.

Related Posts

Improving Harmony's PDF extraction with user testing

Improving Harmony's PDF extraction with user testing

Since we built Harmony, a common complaint has been that it frequently identifies the wrong questions in PDFs. The original algorithm for finding questions in PDFs was a mixture of rule based heuristics and some hand coded logic to look for e.g. lines in the document which begin with numbers. This was very fragile and worked fine on short questionnaires such as the GAD-7, but failed on larger documents. We decided to run a competition with our partner DOXA AI where members of the public could train their own model to extract questions from PDFs.
Harmony at MQ and DataMind Data Science Workshop

Harmony at MQ and DataMind Data Science Workshop

Harmony at MQ and Datamind Data Science workshop On 2 May 2025, Dr Eoin McElroy demonstrated Harmony at the MQ and Datamind Data Science workshop in Deutsche Bank. Eoin’s presentation focused on “Maximising the use of existing survey data: facilitating cross-study research using retrospective harmonization.” The workshop brought together researchers interested in applying novel harmonisation techniques to existing datasets. Eoin explained traditional harmonisation processes and presented a user-friendly guide to the Harmony tool, demonstrating how natural language processing can streamline the harmonisation process.

Signup to our newsletter

The latest news on data harmonisation project.

Please select all the ways you would like to hear from Harmony project:

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our website. We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices.