https://ift.tt/N4SbdKf EVERYDAY INTEGRITY Do Other Fields Make Mistakes Better? Photo by Riccardo Annandale on Unsplash Welcome back...
EVERYDAY INTEGRITY
Do Other Fields Make Mistakes Better?
Welcome back to this series focusing on everyday ethical challenges in data science! My earlier post focused on individual preparation for dealing with mistakes. In the absence of industry norms or strong leadership, much of the burden of acting with integrity falls on individual workers, including junior ones. Does it have to be that way? What role should managers and industry leaders play? Are there other fields with different practices?
In this essay, I discuss ideas and practices from other industries that I believe would help data science leaders promote honesty and transparency. These are linked to the general need for active leadership in AI integrity.
Industry Norms
I have chosen to highlight three fields that might serve as models for data science: actuarial science, software engineering, and academia. I have direct experience working in two of these (software engineering and academia) but rely on published reports and general impressions for the actuarial science. This isn’t a quantitative study; even when I’ve worked in a field, these N=1 or 2 encounters aren’t necessarily representative. However, even my limited experiences show a range of approaches to integrity, some of which are much more effective than others.
If a few jobs in other industries don’t make me an expert in those fields, I can’t claim to know the full range of approaches to integrity in data science organizations. But, I’ve been a few places and observed numerous clients, vendors, etc., and even this moderate amount of experience is enough for me to have seen troubles result from neglect of integrity. These problems ranged from unnecessary confusion to complete loss of credibility.
In addition to my own experiences, I’ve looked for outside information on integrity in data science. In my previous essay, I discussed asking about how to handle mistakes at an AI ethics workshop [1]. My question seemed to come as a surprise to the panel, which focused mostly on algorithmic bias and privacy. In turn, I was surprised that issues of integrity weren’t considered to be on topic for such a group. The issues being discussed were indeed very important, and perhaps it made sense to focus on those. But then, where does a person go for guidance on everyday issues of honesty and integrity?
Two major data science ethics conferences are ACM’s FAccT and AAAI/ACM’s Artificial Intelligence Ethics and Society [2,3]. I searched their 2019–2021 agendas for the words “honest”, “integrity”, and “disclosure”. I found just two matches, neither of which discussed mistakes or lying, but rather user experience confusion and conflicts of interest in academia. Similarly, searches for “error” lead to articles on statistical errors, not human mistakes.
In contrast, such searches on actuarial science sites result in numerous relevant hits. In fact, the code of professional conduct adopted by US-based actuarial societies lists “Professional Integrity” as its first precept [4]. Why the different ethical focuses in data science vs. actuarial science? After all, the fields have many overlaps. My feeling is that actuaries, who have practiced for centuries, have learned from their mistakes. A much newer field, data science seems to be starting from scratch, rather than leveraging wisdom from other specialties.
The Actuarial Model
Actuaries are licensed. Licensing boards set policy and can revoke licenses when ethical breaches occur. There is no license for practicing “data science”. I am based in the US, and I don’t see licensing as likely here (see also [5]). However, there have been efforts to formalize data science in the UK [6]; I am unaware of how these efforts are progressing.
Regardless, I think we can learn a lot from actuaries. First, they openly discuss issues of integrity, set expectations, and provide practical guidance to members [7–10]. They even survey their members about concerns and practices related to integrity and publish the results [7].
The actuarial approach is unambiguous. If you make a mistake, you disclose it no matter what. If someone else makes a mistake or commits a breach, you disclose that, too. If you don’t, you may lose your license and career. Some I have known would dismiss this approach as “naïve”, “simplistic”, etc. However, actuarial science is over 300 years old, with professional licensing existing about 130 years [11]; their approach must have advantages.
I believe such an unequivocal policy is best. Disclosure is the only option that prioritizes the well-being of clients and the public over the comfort of the data scientist.
The Academic Model
In academia, work is performed by numerous, mostly autonomous research groups, with practices highly dependent on the personality of the leading professor. Data science is similar; ethical standards are mostly determined by company culture and individual managers and vary widely among organizations.
In some academic fields, review boards may exist to regulate ethical issues either in a proactive (human subjects committee) or reactive (misconduct board) manner. Similar boards are suggested as a solution to ethical problems in data science [12]. As someone who has worked at very small businesses, I caution that this solution may leave out a large swath of data scientists. Perhaps someday consulting firms, or even rating agencies, will be available. The Algorithmic Justice League/ORCAA, for example, provide review services [13, 14]; however, as far as I know, this process addresses algorithmic bias, not honesty.
Even if such boards were widely available, I feel they are suited for issues where they can be proactive, such as privacy and algorithmic bias, but probably will be less effective for ensuring integrity. The success of any committee is also going to depend on member selection, how safe members feel in their reviews, and how open leaders are to recommendations. I would argue that in academia, truly independent committees are more likely, because members have tenure and some degree of cultural protection.
Of all the fields discussed here, data science most resembles academia, both in current practice and in suggested reforms. However, I think this model has some shortcomings for us. Excepting some research settings, data science is not a science. We don’t develop general theories subject to continuous challenge and refinement. Our results may be applicable only to one company, and never be replicable in any other setting. Plus, our conclusions may be trade secrets, and so not subject to scrutiny. We can move fast, following business cycles, producing a large volume of information that quickly becomes obsolete. Natural correction of mistakes may never occur; we are responsible for the information we produce.
The Software Engineering Model
Many years ago, I worked in software quality control, writing automated tests. There, they planned for mistakes — Quality assurance (QA) was baked into the development cycle. Software engineering also often uses standard channels for disclosing bugs, e.g. software updates and patches.
In contrast, in data science and analytics, the person doing the math or building a model is often solely responsible for quality; formal QA is rare. Not all projects are suited for, say, automated testing, but I personally believe that there are numerous ways to improve our processes. For instance, assertion-based tests on the final, cleaned dataset used for model building (or deployment) are very helpful. Peer reviews and careful (even structured) consultations with clients or subject matter experts are also beneficial.
Regardless of the specifics of a QA process, I feel a QA mindset in invaluable. Mistakes are seen as issues with process rather than as individual shortcomings. Even when I worked with what was considered an elite development team, it was assumed that mistakes would occur. If I’d failed to find errors, no one would have congratulated the developers on their perfection (instead, they’d have assumed I must have missed something). In my opinion, shared responsibility for quality, and acknowledgment that even the best make mistakes, helps people work with less fear and more creativity.
In software, errors are renamed as “bugs” and blame is replaced by process. In addition, they use mature tools such as test harnesses and have established channels for disclosing errors. “Data science” covers a wide range of projects and skillsets, many of which can’t be mapped directly onto software engineering practices. However, I think that we should borrow as many of their tools as we can.
A Hybrid Model
As a new field, data science has the advantage of being able to borrow best practices from a variety of industries with centuries of experience! We can pick and choose might work for us.
If I were asked to design a model for AI integrity, I would use parts of all the fields discussed here. Like the actuaries, we can set high ethical standards. We can openly discuss challenges to integrity, monitor ourselves, and learn from our mistakes. We can be explicit about expectations, while offering practical guidance to workers.
Imitating software engineering, we might improve quality assurance to catch errors before an embarrassing reveal is needed. We can have channels in place for corrections, so we (and our customers) know exactly what to do when an error occurs.
In addition, we can make errors boring — something to be addressed by process instead of individual perfection. We need to acknowledge that mistakes will happen and establish channels for communicating these. In addition, we need to be able to learn from them, instead of just becoming angry at individual imperfections.
As in academia, institutional culture and management style are crucial. Integrity doesn’t happen in a vacuum; our personal values and weaknesses interact with our environment. To successfully apply any ideas from any field, we need active leadership.
Leading for Integrity
What is the role of a leader in the area of integrity? In my opinion, it’s very similar to other areas of leadership. Leaders set expectations and remind people of policies and norms. They remove barriers to accomplishing goals by reducing ambiguity and providing training and mentorship. They set an emotional tone and can provide a good example. They examine, create, and change processes to accomplish their purpose.
There are a great many data science groups with good leadership, positive company cultures, and employees who want to do the right thing. Such organizations might not discuss issues of integrity directly, but the general climate is conducive to good ethics. These groups are fine now and most of them will be fine in the future. Are these effective teams “good enough?” Or, is it necessary to be more purposeful about integrity?
In my opinion, data science teams with pleasant cultures and good people, but which fail to directly address integrity, are taking a chance. A lack of specific attention to integrity leads to ambiguous situations, which are higher risk for self-justification or minimization. In addition, groups don’t operate in a vacuum. Employees may move company to company, and clients who have had bad experiences need reassurance from new vendors.
I’ve seen examples of how not to lead for integrity, and these are also helpful in understanding the importance of purposeful action. I’ve seen quality control totally neglected, followed by bewilderment when issues occur. Best practices for integrity are often never discussed, and managers don’t set expectations with new employees. When an employee makes a mistake, some managers become angry or punitive (I’ll discuss psychological safety and related issues in future essays). Worst of all, I’ve seen leaders decide not to disclose errors affecting external clients — yet these same leaders expect full transparency from their employees.
Both benign neglect and toxic leadership endanger the credibility of the field, in addition to putting clients and the public at risk. In my opinion, lack of attention to integrity also tends to come back to bite leaders, leading to project failures, loss of clients, and unhappy employees. I believe that leading for AI integrity involves being explicit about best practices. We must be unequivocal about the value of honesty and the necessity of disclosure.
One of the most crucial points of ambiguity in data science integrity is exactly how and when to disclose errors. Every person doing data science work should understand what to do if they find or make an error, and leaders need to set expectations with other stakeholders.
Clients often don’t ask about how errors will be communicated when they engage an analytics firm. In turn, the data science organization doesn’t bring up this issue, so that when a mistake happens, we aren’t sure if or how to inform the client. It’s almost as if mistakes are underworld demons, which must not be mentioned lest they are summoned. But this lack of conversation leaves a lot of room for prevarication. Maybe the client isn’t using it anymore, maybe they wouldn’t want to be bothered with an issue I’ve convinced myself can’t possibly be that important, maybe the damage is already done.
Similarly, dashboard user interfaces are often designed with no way to inform users of past, or even present, errors; I’ve seen a couple occasions where dashboard errors caught a team entirely off guard. With no plan in place, they scrambled to take a website down “for maintenance”, or just left it up assuming no one would notice before a fix was in.
It seems that there are standard channels for disclosure of errors in other professions. In journalism, a newspaper might print an erratum. In software engineering, they use patches, updates, and even press releases. In data science, channels for error disclosure might vary project to project, or be informal or formal, but should exist and be part of project planning.
Final Thoughts
When leaders fail to address issues of integrity, the burden will fall onto individual workers. They are expected to know that they should disclose an error, to whom, and to do this promptly and diplomatically. Often an organization provides little to no QA support, code reviews, etc., leaving that also to the individual practitioners.
Who are these workers? Unsurprisingly for a newer field, many data scientists are relatively inexperienced. The Stack Overflow survey lists Data Science and Machine Learning engineers having fewer working years than any other type of developer, other than student [15]. Kaggle’s survey found that over half of data scientists are under age 35 [16]. According to Business Insider, the number of data science programs in the US nearly quadrupled in six years [17], and much has been written about a difficult job market for these numerous new graduates.
It is just not right to place primary responsibility for integrity on such a vulnerable, inexperienced workforce. All of us with a few years on must take at least some responsibility; moreover, we must demand better from all levels of leadership. Luckily, other fields provide a wealth of ideas and multiple options for developing best practices in this area.
References
[1] V. Carey, AI Integrity: Planning Ahead to Do the Right Thing (2021), Towards Data Science
[2] ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT) (2021), Website
[3] AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (2001), Website
[4] American Academy of Actuaries, Code of Professional Conduct (2001)
[5] A. Lyskov, Should Data Scientists Be Licensed? (2019), Towards Data Science
[6] B. McKenna, BCS, Royal Statistical Society, Alan Turing Institute combine to cement data science professionalism (2021) ComputerWeekly.com, Jul 23
[7] American Academy of Actuaries, Key Ethical Concerns Facing the Actuarial Profession: Perceptions of Members of the American Academy of Actuaries(2015)
[8] A. Lacroix, Ethics and Professionalism among Actuaries (2017), Seeing Beyond Risk, Canadian Institute of Actuaries Document 217062
[9] R. Kutikoff, The Ethical Actuary (2014), Plan Consultant, Summer
[10] L. Bloom, Errors and Omissions: What to Do When You’ve Made a Mistake (2012), Plan Consultant, Summer
[11] Society of Actuaries, Historical Background (2021)
[12] R. Blackman, If Your Company Uses AI, It Needs an Institutional Review Board (2021), Harvard Business Review, Apr 1
[13] Algorithmic Justice League, Request Algorithmic Audit, Website (2021)
[14] ORCAA, Website (2021)
[15] Stack Overflow, 2021 Developer Survey (2021)
[16] Kaggle, State of Machine Learning and Data Science 2021 (2021)
[17] J. C. Oliver and T. McNeil, Data science jobs are growing, but not all students are given ethics trainings before starting their careers (2021), Business Insider
AI Integrity: Leadership Lessons from Other Industries was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
from Towards Data Science - Medium https://ift.tt/rDSxAOJ
via RiYo Analytics
No comments