Anyone who was at the RSA Conference this year would think that machine learning, and its close cousin artificial intelligence, were the saviors of our industry. However, last week we got a wakeup call after Microsoft introduced Tay, its artificial intelligence Twitter bot, to the world.
Unfortunately, the experiment backfired. Microsoft shut down Tay in less than a day after the bot, which was designed to mimic the casual speaking habits of a teenage girl, turned into that uncle who’s so racist he doesn’t get invited to Thanksgiving any more.
This entire experiment contains some interesting lessons for folks like us who are incorporating machine learning into our products.
-- Just because it’s “normal” it doesn’t make it right. Just like Internet chat rooms, comments pages and Twitter, corporate computing environments are filthy places, rife with all manner of digital pond life. Baselining “known good” in a real-world, diverse computing environment is really tough.
-- When it goes wrong, it goes WRONG. If you’re not careful, machine learning can afford a level of false security, and one wrong turn can result in unanticipated consequences of massive proportions. Over-reliance on this technology can be disastrous, so being able to spot problems early and change course is essential.
-- Even the most sophisticated systems still need significant human intervention. Despite all its tweaking, Microsoft shut down Tay within a day. A bit like your long suffering cousin who promised to give your uncle a ride home, and to spare everyone’s blushes suddenly invented an early Thanksgiving evening engagement he needed to attend. Machine learning is only as good as the supervision, and supervision can be extremely onerous. My colleague Yoav Orot made a similar point in a recent blog post looking at how machine-learning technology can’t grasp a person's intent. Human actions can only be understood by another person, not an algorithm.
That’s not to say machine learning can’t be a useful tool. When you take a brand spanking new laptop out of the box, provided you have decent control over your supply chain, you can have a reasonable expectation that what happens on that machine is “known good.” Obviously, over time as end users start a) doing real work, and b) playing fast and loose with a few minor corporate policies, then things will get more complicated. However, if you see big changes in behavior then this is definitely an indication that something is wrong.
Alas, once again we learn that there are no silver bullets in the security industry.
Paul Stamp is the Director of Product Marketing at Cybereason.
