TrustyAI – an open source project looking to solve AI’s bias

Rebecca WhitworthBy Rebecca Whitworth, Manager, Software Engineering at Red Hat. 

Artificial intelligence (AI) is an exciting area of technical development. As the tools and methodologies advance, many organisations are looking to use AI to improve business efficiencies, bring innovation to customers faster, gain actionable market insights and more.

However, the rush to put AI in place without always knowing what it can be used for, or how to use it, can lead to problems with the systems and the data itself. We have heard many stories of when AI makes the “wrong” decision due to built-in biases, and in some cases, the outcome can be life or death.

This is a challenge that open source project TrustyAI – built on Red Hat OpenShift and Kogito – is looking to address.

AI gone wrong

AI is essentially a set of priorities and decisions a system has been programmed to follow. And because humans are responsible for that programming, our flaws become the system’s flaws. This happens in two main ways. One is cognitive bias that comes from the person, or people, who train the data model and build bias into the AI logic. The second problem is a lack of data, or a lack of diverse data. For example, we’ve seen cases of datasets that are dominated by white males, meaning AI trained on this data filters out anyone who doesn’t fit these characteristics.

There is a growing amount of research about the transfer of bias from humans to AI. As AI becomes more prevalent, it is being used to decide critical life moments like whether you get into college, get a mortgage, qualify for a medical operation, and even for determining a prison sentence, which makes the negative outcomes easier to spot.

The data science community is very aware of the need to stop systems perpetuating bias, and all the big tech companies are now looking at this. IBM’s AI Fairness 360, Google’s “What-If Tool” and Amazon SageMaker Clarify all go to show how significant and competitive this field has become.

Identifying the problem

If you aren’t proactively looking for bias, you may not notice if the AI you are using for hiring is selecting people based on their gender and name instead of their technical skills and experience, for example. There hasn’t been a simple and reliable way to test the balance and accuracy of an AI model, so the first time a business knows it has an issue is when it gets some negative PR or someone brings a case against it. This black box opaqueness has also become a legal risk with the advent of the General Data Protection Regulation (GDPR), where anybody in Europe has the right to ask for their data and how it is being used.

TrustyAI is a standalone piece of software that plugs into any AI application, regardless of what environment or technology it’s running on. It introspects the system, looking at all the various inputs, and maps how these influence the outputs. That analysis is served up in easy-to-visualise charts, which make it clearer if any biases exist. The user can alter the values to see the effect of different criteria.

For example, with a rejected mortgage application, you’d expect a different outcome if the applicant had a higher credit rating or earned more money. But if TrustyAI finds that by only changing the applicant’s ethnicity or postcode you arrive at a different outcome, then it’s clear the system has bias. We call this Explainable AI — XAI for short.

TrustyAI in action

TrustyAI has been built with Kogito (an open source, end-to-end business process automation technology) and Red Hat OpenShift. So it is cloud-native and can be deployed to any system running on any environment. It is written in Java and integrates with Python, the language data scientists generally use, so a user only needs to open up a notebook and apply library functions to their datasets. There’s no reconfiguration required.

Open source has been critical to TrustyAI. Tapping into open communities has enabled us to develop at a faster pace. For example, KIE (Knowledge Is Everything), an open source community for business automation, has been a brilliant partner. Its repository hosts the TrustyAI code and its members continually perform testing, offer fixes and suggestions and share their datasets. Without this collaboration (our in-house project team is myself, three developers and one architect) progress would have been much slower.

We’re really excited about the impact that TrustyAI can have. Reputational management can make or break a business, and there’s a real clamour to mitigate the risks of AI. Understandably, businesses don’t want to slow down their adoption of AI, and TrustyAI helps give them the confidence to push forward, while also prioritising transparency and accountability.

As a project team, we are looking at how to apply TrustyAI across Red Hat’s portfolio to add strategic value — both internally and for our customers — while continuing our research on all things AI, from cognitive bias and ethics to developments in data science and algorithmic best practices.

About Red Hat

Red Hat logoRed Hat is the world’s leading provider of enterprise open source software solutions and services, using a community-powered approach to deliver reliable and high-performing Linux, hybrid cloud, container, and Kubernetes technologies. Red Hat helps customers develop cloud-native applications, integrate existing and new IT, and automate, secure, and manage complex environments. Follow Red Hat on Twitter: @RedHat

Ada Lovelace: A growing legacy

While Ada Lovelace lives on as an inspiration to women working and researching in STEM subjects through the activities of Ada Lovelace Day, her impact can also be seen in the literature. Using tools available from Digital Science, Simon Linacre looks at the ever-increasing amount of research surrounding one of science’s most fascinating figures.

Firstly, a confession: before starting work for Digital Science in early 2022, I didn’t know who Ada Lovelace was. I knew the name, and may have known she had been a scientist of some description, but I had no idea what she had achieved or when she had achieved it. Put it down to my preference for the arts in my education, a patriarchal society or just sheer ignorance, but I had no idea what an inspirational figure she was. Of course, now I know a lot more having worked with Digital Science in its support of Ada Lovelace Day, but I wanted to know more. As someone who has worked in scholarly communications for most of their career, what does her legacy look like in the current literature?

At first glance, it is pretty significant. According to the Dimensions linked database – which covers 131 million publications – there are over 15,000 mentions of ‘Ada Lovelace’ in those publications, as well as 74 policy documents and 29 patents. There are even mentions in eight grant applications. Looking at who is doing the mentioning, it includes some major institutions in scientific research, including Oxford, Cambridge and University College London.

Having said that, most publications originated in the US and not the UK, which shows that her influence has not been limited to her country of birth.

This wider influence is also in evidence when it comes to research categories. While it is unsurprising that the most common research category with mentions of her name is Information and Computing Sciences (716 mentions), not far behind are Philosophy and Religious Studies (486), Language, Communication and Culture (271) and Human Society (240). Lovelace has clearly had quite a broad spectrum of influence, far outside of her ‘home’ of computing sciences.

And the influence appears to be growing. As you can see from the chart, despite a lull around 2018, interest in Ada Lovelace has been growing steadily in recent years, with some acceleration more recently when we look at total outputs of publications that mention her name.

Graph showing the increasing number of mentions of Ada Lovelace in academic publications

Perhaps one reason for her popularity is the proportion of articles that mention her being open access, with 79% of them being free to read online.

Focusing again on the present day, Ada Lovelace has also made somewhat of a splash online. When we look at Altmetric data, we see that the increase in mentions trackable on the internet have also seen rapid growth since 2016, the year after her bicentenary, and especially in the last couple of years. In 2021 alone (see below) there were over 8,000 recorded instances, including 6,662 Twitter mentions, 1,028 Wikipedia mentions and 312 news items.

Bar graph showing increasing mentions of Ada Lovelace in Altmetric's data.

So, who was Ada Lovelace? These days, she is best known for being the first person to publish what would today be called a computer program.

Throughout her childhood she was fascinated by machines, and at the age of 17, she was introduced to the engineer and inventor, Charles Babbage, and his general purpose mechanical computer, the Analytical Engine. Lovelace and Babbage became life-long friends, and Lovelace came to understand the Analytical Engine in depth.

When Italian mathematician Luigi Menabrea published an article about the Analytical Engine in French, she translated it into English, correcting some errors as she went. Babbage suggested that she added her own footnotes, which she did, tripling the length of the paper.

In these footnotes, Lovelace wrote a set of instructions for the calculation of Bernoulli Numbers. Although Babbage had written fragments before, her more elaborate program was complete and the first to be published. She also speculated on the future capabilities of the Analytical Engine, suggesting that it could be used to create original pieces of music and works of art, if only she knew how to program it. She recognised the enormous potential of machines like the Analytical Engine and her vision bears a striking resemblance to modern computer science.

Lovelace’s work was truly ground-breaking and her achievements become even more impressive when one remembers that she was working from first principles with only Babbage’s designs and descriptions to guide her and no working computer to tinker with. Yet, the importance of her paper was not recognised until Alan Turing’s work on the first modern computers in the 1940s.

As we can see from the data, Lovelace is now a widely cited and mentioned mathematician who is increasingly influencing a broad range of STEM and social science areas. Not only does the legacy of Ada Lovelace live on, it has never been bigger or more important.

About Digital Science

Digital ScienceDigital Science is a technology company serving the needs of scientific research. We offer a range of scientific technology and content solutions that help make scientific research more efficient. Whether at the bench or in a research setting, our aim is to help to simplify workflows and change the way science is done. We believe passionately that tomorrow’s research will be different – and better – than today’s. Follow Digital Science on Twitter: @digitalsci

The diverse world of women in tech

Michelle Szaraz, delivery lead at dxw, talks about what makes a woman a ‘woman in tech’ in this post from 2021. 

Michelle SzarazI’m not a super techie person and yet I spend at least 35 hours every week working in technical teams, designing and building digital public services. Beyond my day job as a Delivery Lead at dxw, I mentor, write, and speak about a range of business, career, and female empowerment topics, including women in tech.

I certainly consider myself to be a woman in tech, however, this wasn’t always the case. Until recently, I used to believe women in tech were exclusively those with a STEM (science, technology, engineering, and mathematics) background. I’m not the only one to feel this way. There’s still a lack of clarity on who women in tech are, alongside a limited representation of their various experiences.

That’s why I decided to take advantage of Ada Lovelace Day and dedicate some time to sharing just how diverse the world of women in tech can be. I want to help push the inclusion of females in tech a step further, and make sure all women feel included regardless of their personal or professional backgrounds.

We need to make the technology industry, workplaces, and society in general more inclusive, and therefore happier and more productive places.

Am I a woman in tech?

Following my earlier career in a range of roles, companies, and sectors, from retail to international development, I jumped into the world of tech. It’s been a couple of years now, but it’s only lately that I started to see myself as a woman in tech. Why is that?

For a while, I thought my non-STEM education and professional experience meant I didn’t belong as part of the “women in tech” group. It wasn’t a completely unsupported assumption. Many activities and initiatives to increase female involvement in tech are centred around attracting women to STEM related qualifications and positions across the industry. This is applaudable (and overdue) because until not long ago, those opportunities didn’t really exist.

But what about all the females working in tech that don’t have a STEM background and/or don’t work in technical roles within the industry? Are they also women in tech? It’s a very clear yes from me. But we’re not always clear about that, both as individuals and institutions.

Only last week, for example, I joined a launch session for a “women in tech” mentoring programme. An expert in marketing asked whether she could be involved as a mentor because she didn’t have a technology background. She was not only reassured she could add significant value, but also surprised to find out she was in fact a woman in tech. The group reminded her that most marketing now happens online via social media and is “quite techie”.

So who are the women in tech?

I come across situations like the one above fairly regularly. I consider this lack of clarity about who women in tech are, a blind spot in the opportunities and support offered to promote female inclusion across the technology industry. A good starting point in tackling this is bringing attention to the diversity of experiences women in tech have.

I’ve been lucky to meet and hear stories from women in tech that come from a great range of backgrounds. And yet, until recently, I assumed there were 2 main female (stereo)types in tech – the STEM women in tech roles and those without a STEM background working in “non-tech” positions (like myself).

Pretty straightforward, right? At least it was, until I heard one of my colleagues speaking about her education and early career within the STEM field, before transitioning into the “non-technical” role she currently holds. After the initial, “Wow, mind blowing!” moment, it got me thinking. Just as women with non-tech backgrounds can train up to take on and grow into high level senior technical positions, it’s equally possible for females to shift from STEM roles into less technical ones throughout their careers.

In short, women in tech really do have surprisingly varied and unique career paths and experiences. And we don’t talk enough about that. We should!

What’s next?

Personally, I’d never considered that I could one day work in the technology industry until I got hired for a role in a digital technology innovation centre through a recruitment agency. Had I known what the position or the company was, I probably wouldn’t have applied. Why? Because I would have taken it for granted that I needed a STEM background to succeed. And in doing so, I would have missed out big time.

By being thrown into the tech world, I proved my own unconscious assumptions wrong before even realising I had them.

Sharing our individual stories and experiences as women in tech with different professional and personal backgrounds can make a huge difference to other females, both those already within the technology industry and those who might work in it one day. Beyond offering the, “Oh, it’s not just me then…” kind of realisations, it also helps us uncover where we need to pay extra attention and make more effort to progress.

And that’s how I’d like to wrap this post up – by encouraging not only my fellow dxw colleagues, but all other women in tech to celebrate Ada Lovelace Day by sharing their stories and the journeys that led them to a career in tech. I can’t wait to hear them!

About dxw

dxw logodxw is a leading employee-owned digital agency that works with the public and third sectors. Our employees are based across the UK. We get together for work and social things at our Leeds HQ and London hubs. We’ve made it our job to fix some of the really difficult stuff in government. We’re proud to have supported critical national infrastructure through the pandemic, including NHS England and Homes for Ukraine. We help organisations shape their strategy and design, build and run better digital public services. We work in the open wherever we can, you’ll find the way we work in our playbook and our code on GitHub. We became employee owned last year and we love it! It gives us the freedom to work on projects we’re passionate about and where we know we can add real value. Find out more at www.dxw.com. Twitter: @dxw

Understanding micromanagement

Cynthia Sanchez, a senior technical project management at SUSE, explains how to spot micromanagement, why it’s problematic, and how to stop yourself become a micromanager.

This video was provided by SUSE, one of Ada Lovelace Day’s generous sponsors.

About SUSE

SUSE logoSUSE is a global leader in innovative, reliable and enterprise-grade open source solutions. We specialise in Enterprise Linux, Kubernetes Management, and Edge solutions, and collaborate with partners and communities to empower our customers to innovate everywhere – from the data centre, to the cloud, to the edge and beyond. SUSE puts the “open” back in open source, giving customers the agility to tackle innovation challenges today and the freedom to evolve their strategy and solutions tomorrow. Follow SUSE on Twitter or LinkedIn.

Advice from a frontline manager

By Katerina Arzhayev, SUSE’s Leader of COO Initiatives

At the age of 25, I began managing people for the first time. My new direct report was just starting her full-time career. Seemingly overnight, I became responsible for her professional and personal development. As an individual contributor, my success was based on my performance. Now, my success had a component outside of my control: someone else’s performance. So, I did what anyone would and sought council. The best advice I heard came from a surprising source – my nineteen year old sister who had been people managing for several years as a supervisor in the service industry.

I was told that if I worked hard, studied often and performed well, one day I would manage a team. My sister didn’t get the memo, so three months into her role as a barista she volunteered for a Supervisor role to manage a team of peers – people she laughed and joked with, 17 and 18 year olds she considered friends. Overnight, she became a boss.

“I stopped treating my coworkers as if they are my family” she said. “I forced myself to have the mindset that you’re supposed to have in a service job, which is ‘I am here to work’. I had the mindset, ‘If I don’t like you, I won’t like you’, but I am a shift lead. So even if I am mad, I am still nice to them. I got a more realistic view of my job.”

This important realisation helped her understand that difficult conversations and critical feedback are not personal – not to her, as the giver, and not to the employee, as the receiver. Her communication of company processes and policy did not make her a bad person, no matter the side glances or hushed complaints.

Yet, here I was, afraid of delivering any feedback that could be perceived as negative for fear my relationship with my new team would be damaged. How would my entire personality, built on being liked, handle a potential conflict?

“When I manage, I remove myself from the situation,” she said. “If I am managing someone, I put their needs and perspective first. It’s not about how I feel. If, all of a sudden, they get emotional, I don’t take that personally. I am the manger.”

Being a manager is a different kind of role, she explained, but it is not one where you need to change who you are, or who you want to be.

“I know all the employees on a personal level. I try to customise my approach based on the person, because I know them. I did my research. I am not a people pleaser, but I am not shy. I am not going to stay quiet if someone needs managing.”

Yet, this takes time. Step by step, we can build those relationships. My sister’s rules are simple and by following them, both her managers and her team know who she is and what kind of manager she aspires to be. This may seem like such basic advice, yet all of us need the :

  1. Don’t talk about me with other employees, or about other employees with me. If you have something to say, talk to me in private, say it to my face. Don’t say it to other people whom it doesn’t concern. Don’t be afraid to address the problem head on, when it arises. Do not let things fester. Avoiding conflict causes longer, and more drawn out, conflict.
  2. Don’t set me up to say things. Be direct with your questions and comments, instead of leading and secretive. Be upfront, don’t play your cards. Don’t manipulate the situation. Through open conversation and an honest desire to resolve things, we can all move forward.
  3. Don’t focus on me. Micromanaging sows dissent and turns off my creative desire to be and do better. When I know everything is set, I stop being proactive. I stop demanding more from my team. I am in this role for a reason, I was promoted for a reason, I manage people for a reason. Provide me with support and guidance, but also let me take risks and fail.

Note: The content of this interview was in the scope of primary research conducted by Katerina Arzhayev with the goal of describing the average career journey of women, and identifying and documenting their best practices for managing and being managed.

About SUSE

SUSE logoSUSE is a global leader in innovative, reliable and enterprise-grade open source solutions. We specialise in Enterprise Linux, Kubernetes Management, and Edge solutions, and collaborate with partners and communities to empower our customers to innovate everywhere – from the data centre, to the cloud, to the edge and beyond. SUSE puts the “open” back in open source, giving customers the agility to tackle innovation challenges today and the freedom to evolve their strategy and solutions tomorrow. Follow SUSE on Twitter or LinkedIn.