Real World Machine Learning with Susie Sheldrick #APIDAYSAU

I’m at the API Days conference, and one of the first sessions of note was Deep Learning: Real World Applications with Susie Sheldrick, which explored some of the practical real-world challenges related machine learning, based on experience.  I also caught up with her after the session where we expanded on some of curlier questions.

Quick Context: 30 Second Intro to Machine Learning

Susie kicked off with a simple diagram that sums up what machine learning is in comparison to traditional applications:

DeepLearningSusanSheldrick

Machine learning partially turns this model on its head: the solution is able to “learn” its own rules (through training its internal rules model) at much greater scale than some person/team coding them by hand.  So, rather than feeding data and manually created rules into an solution, simply train the solution to produce its own rules.

The Chaser

This nice intro kicked off a mental train of thought for me: in practice the more complete solution probably looks something like this:

DeepLearningAdrianKearns

The end goal is still to build an solution that provides the answers users were seeking, we’re simply using machine learning to help out with the rules.

Devil in the Detail

That all sounds wonderful on paper – or in ivory-tower pixels – but, as should be no surprise, the real world is not so straightforward.

Of critical importance:

  • Understanding the problem you’re trying to solve.
  • Gathering the right data to train the model.

This is much easier said than done, it transpires that:

  • It’s all too easy to inadvertently train bias into the rules model.
  • Tracing exactly how the AI made a specific decision actually turns out to be really hard.

Whilst the second point has obvious implications for developers and testers, both points combined have massive implications for your legal teams, anyone who considers themselves ethical (like you, right?), product owners and anyone at the receiving end of a machine determined decision.

Bias

Susie gave some examples of unexpected and undesirable bias ending up in rule models, such as one experiment that determined prisoners eligibility for parole.  It turns out that the model significantly favored granting parole to white prisoners and was relatively much less favourable to prisoners of colour.   In contrast, in terms of parolees reoffending – the actual results were the exact opposite of the bias.

It turns out that the information used to train the model was “correct” but only in the sense that it faithfully transposed the bias already inherent in the legal system, against people of colour.

True Representation

A related issue isn’t so much of bias in the data, but of bias stemming from an absence of data.  Once more issues of race come to the fore; this time it was a passport application solution that told an Asian gentleman his submitted photo “did not meet our standards” because he was “asleep”.  As you might be able to guess, the model had obviously not been sufficiently trained with data that faithfully represented the entire user base, and therefore could not correctly handle non-european facial features.

Just to be crystal clear, the technology is more than capable of correctly handling a wide range of cases, nuances and subtlety – including racially based facial features.  The actual issue is the correct training of the model – meaning it’s critical to gather the right data, data that covers the entire spectrum of cases.  Not to mention testing and monitoring the behaviour of the solution.

Building an AI Solution: Custom or OOTB?

If you’re about to embark on a project that involves machine learning, one of the practical questions you’ll come up against is whether or not you can use an Out-Of-The-Box (OOTB) solution, or need to custom build something.  Susie’s discussion here was mostly in reference to the rule models specifically.  If you want a model capable of identifying cats in pictures online for you meme generator – you’re in luck, but if you need to correctly identify something more obscure, or more specific, you may have to build this model yourself.  Which is why the stuff above about bias is so important, because you’re going to have to navigate that minefield yourself.

Further Questions

Our chat after the session was very stimulating; a couple of the more curly questions that our conversation provoked were:

How to identify, and test for, unexpected bias?

The obvious ethical reaction to all of this is “great, let’s ensure we keep unwanted bias out of the model and our solution”.  What is much less obvious is how to do that.

Were the team behind the parole example conscious of the bias in that solution?  Let us assume they weren’t aware of it – in such a situation how would they (or you) identify that bias, and in addition, having established an operational solution how would you ensure none was introduced?

This is where, for me, machine learning is like a lens that amplifies human behaviours and bias.  It has the potential to help expose them, but how clearly, how soon, and at what cost?

How will your model react in the event of change over time?  I.e. if there is a fundamental shift in the (data) foundations on which the model was originally conceived and trained?

For example, Google is looking at moving back into the Chinese market, despite pulling out some years ago due to human rights concerns.  Hypothetical example: let’s assume that they have machine learning models built up, based on the data they currently have access to – i.e. does not include China’s current population of 1.3 billion.

What would happen if 1.3 billion Chinese people suddenly have access to a Google solution that is backed by a rules model that was not trained with them in mind?  Sure, Google’s data should be a fair representation of their current global user base, which will include Chinese – but wouldn’t adding 1.3 billion people potentially shift the model?  How will it react?  Will the responses it provides be biased against the new user population because hitherto they were not expected by the model?  Will the model be able to adapt over time, and if so how long will that be?

 

Please note that this post is based on rapidly scrawled notes in session and my recollection of subsequent discussions – my accuracy should be reasonable but may not be perfect.

References:

Fireside Chat with Zheng Li, VP of Product @ Raygun – Product Tank Wellington MeetUp

The Product Tank Wellington meetup ran a “fireside chat” last night with Zheng Li, who is currently VP of Product with Raygun – a Wellington-based company, currently on loan to the U.S.

The conversation covered her career path to Product via UX, advertising, championing women in tech and passion for business, as well as delving into specific topics with being a product person.

Here are the key takeaways I jotted down, which I’ve tried to organise by topic…

Career Path

Zheng gave us a neat little story about how she started out (in a sense): a classic tale of taking something that nobody else wanted to do and absolutely nailing it.

The task was designing banner ads for TradeMe.  She obviously attacked her self-imposed challenge with passion and drive (significant keys to success on their own), but I also noted that:

  • She formed a loose multidisciplinary team which (I think) included people with knowledge and access to data analytics and marketing folks.
  • Was data driven – each time she/they ran a new design, they would analyse the data to see what was working and what wasn’t, and think about why that was the case.

The other factor which she used to her advantage was being able to iterate at an appropriate speed – which was obviously supported by the data she and her team had access to.

Some pretty obvious takeaways there, a key one for me would be about being data driven / enabled > implication: you need to have the data.  As a data architect colleague of mine once said: before doing any data design, you must first think about what questions you will want to ask your data.

Other stand-out points around career path included:

  1. Turning weaknesses into strengths, by using them as differentiators.  The context for this was around credibility.
  2. Follow your passion.  Zheng laughed in response to a question – someone asked something which inferred she had planned her career out; she said that in retrospect her career may look like it was planned but the reality at the time was anything but.  Her response to challenges was to consciously seek out ways of addressing these – which in her case frequently included training courses, which she collectively found effective (I think for one particular area she did 7 different courses).
  3. People want to work with people they like and trust.  Zheng spoke of this in reference to relationships between companies, but it’s obvious from her perspective that this is based on interpersonal rapport.  It’s not hard to see this concept also applying at a personal career level – something I can attest to having also experienced it first-hand.

Another key career theme Zheng had was based on “that venn diagram” – meaning the three overlapping lenses in Design Thinking which cover business/viability, technology/feasibility and people/desirability.  The specific terms she used might have been a little different, but for me the connection was pretty clear.

Her basic advice was to become proficient and confident in any two of these lenses; although that seemed to be somewhat tempered with her other guiding principle of being customer focused – which suggests the business/viability and people/desirability lenses.

“Product” Means Being Close to Customers

This was one of Zheng’s key themes.  Part of this was getting out and talking to customers, which is critical.

It was interesting to hear of her experiences using product “management” (my term, not hers – can’t recall exactly what she called it) as a selling tool.  The basis for this was:

  1. Selling the value of the product, not the product.
  2. Establishing a 1-on-1 rapport with people, and understanding what kept them up at night.
  3. Taking the time to really understand that problem from different angles.

As far as point #3 goes, that meant engaging with different people in the organisation to understand the problem from their perspective: technical, marketing, sales, etc; this obviously links back to the three lenses of design thinking mentioned above, and being close to customers – all good sensible product management stuff.

We can also expand this theme out “customers” to “people”.  In her experience, product management is more about being people-based than technology based (this was mentioned in reference to a technical product for developers).

There was also a leadership angle: for her leadership was about aligning the purpose of her staff to the purpose of her business.  The implication here is to talk with the people on your team and really understand what drives them and where they want to go with their career.

A Quick Note on Persuasion

If you want to persuade someone (such as your product manager – if you’re a tech working on the product, and you have a pet feature you want to add), you need to two things:

  1. Speak in the language of the audience.
  2. Back it up with data.  This could be qualitative such as customer feedback, or quantitative data showing conversion rates.

Producty Bits

Dealing with Product Debt

Something I really liked was how she addressed debt – debt in the sense of technical debt, and even marketing debt, and so on: things which worked but could work better and had gotten to the point that they were affecting the bigger picture.  She referred to it (I think) as the “99 issues” or “99 problems” story.

  1. They got all the issues and logged them into Jira – meaning that they got it all out into the open.  Not just development/technical debt, everything.
  2. Presumably some sort of sizing and prioritisation work took place.
  3. They then knocked off a number of the items, reducing the overall debt.

The way she spoke seemed to indicate this was an annual event – which didn’t happen every year.  Bit of a spring-clean, I guess.  Zheng didn’t call it out specifically but based on her other comments I presume space in the teams capacity / product roadmap was allocated to this work.

Another interesting idea which occurred to me as she described this was the technique that Agile / Scrum teams sometimes use, whereby they adopt a sprint goal – something non-deliverable – that they want to improve during the course of the sprint/iteration/timebox.  Zheng didn’t explicitly say that was what they were doing but the idea seems relevant.  Zheng, if you ever read this I’d be interested to know if that concept was one you consciously used or were aware of.

Roadmap

Items on a roadmap (i.e. the implied promise / expectations set) should be based on two things:

  1. The teams capacity to deliver them.
  2. Evidence that a given feature is wanted by customers.

Pushing Back

Don’t be afraid to push-back.  If a customer requests a feature (for example) that  is outside your roadmap and/or ability to deliver then be wary of following the money.

This definitely fits with my experience; I tend to think that at a inter-business level or interpersonal level, the relationship needs to be built on mutual trust and respect – if the other party does not reciprocate then they’re probably not someone you want to be dealing with.

Zheng gave two examples:

  1. A major multinational effectively tried to bully their 50 wanted features on top of Zheng’s existing product roadmap – “you want our business or not”?  To have done so would have caused massive chaos within the company, affecting product delivery and so on.  Zheng counter-proposed a different approach which she and her teams could sustain.  The multinational rejected the offer and went elsewhere – only to return months later, accepting Zheng’s proposals.
  2. Another major company approached Zheng with features (she didn’t give specifics but I think we can guess their approach was more reasonable and more adaptable).  Zheng recognised that some of these features would be great differentiators for their product, so (presumably) some changes were made to the product roadmap and the featured added – in essence Zheng followed the money,  but did so because there was further advantage than just the money.

Final Thought: The Iron Triangle

At one point Zheng told an anecdote about a developer talking with her about code quality.  I forget the story but it reminded me of the the old “Iron Triangle” or project management triangle – the one that is made up of scope, quality and cost (or some similar combination; cost and time obviously being closely related).  The model effectively states that you can control any two; the implication being that if you nail people down in terms of scope and cost (or time) you have no control over quality.

I asked Zheng if she was familiar with that model and how she approached it.  Her answer wasn’t as clear-cut and direct as I would have hoped (which is not a criticism – having presented publicly I know how hard it is to provide an off-the-cuff answer that is cohesive and concise), but seemed to boil down to this:

  1. Her first substantive reaction was to discuss scope and features, so I would guess that this is her first priority.  This would align with her other comments that put great importance on being close to the customer and understanding their needs.
  2. Her second substantive reaction was to discuss product roadmaps, specifically in reference to their timing and how they are used as the basis for cross-team coordination (marketing and so on), so I imagine time would be her second priority.

By default this would leave quality to manage it self; but we shouldn’t forget the “spring clean” approach, whereby random items of debt (arguably involving quality) can be addressed in a structured way.

 

 

Use a Single Object Reference Globally in a UWP Application

Sometimes you might want a single object reference that can be accessed from any page within your application.  Creating an instance of it on the App using App.Current may be appropriate.

In this example we’re using a string, but you could easily use a custom object of your own creation.

In the App.xaml.cs file:

sealed partial class App : Application
{
  public string SomeValue;

  /// <summary>
  /// Initializes the singleton application object. This is the first line of authored code
  /// executed, and as such is the logical equivalent of main() or WinMain().
  /// </summary>
  public App()
  {
    this.InitializeComponent();
    this.Suspending += OnSuspending;

    // Do any initiation work here - just don't break the app 
    // or do anything that will slow it down too much.
    SomeValue = "New Value";
  }
}

To use in your code, e.g. on some page somewhere:

public sealed partial class SomePage : Page
{
  App _AppReference = App.Current as App;

  void SomeMethod()
  {
    _AppReference.SomeValue = "New Value";
  }
}

 

 

Using the Clipboard in UWP

How to Copy to Clipboard (Text String)

using Windows.ApplicationModel.DataTransfer;

...

dataPackage = new DataPackage();
dataPackage.RequestedOperation = DataPackageOperation.Copy;
dataPackage.SetText(text);
Clipboard.SetContent(dataPackage);

 

How to Paste from Clipboard (Image/Bitmap Data)

using Windows.Storage.AccessCache;
using Windows.Storage.Streams;

...

DataPackageView dataPackageView = Clipboard.GetContent();

if (dataPackageView.Contains(StandardDataFormats.Bitmap))
{
  IRandomAccessStreamReference imageReceived = null;
  imageReceived = await dataPackageView.GetBitmapAsync();
  if (imageReceived != null)
  {
    using (var imageStream = await imageReceived.OpenReadAsync())
    {
      var bitmapImage = new BitmapImage();
      bitmapImage.SetSource(imageStream);
      imgImage.Source = bitmapImage;

      dec = await BitmapDecoder.CreateAsync(imageStream);

      // https://docs.microsoft.com/en-us/uwp/api/windows.graphics.imaging.bitmapdecoder
      // https://docs.microsoft.com/en-us/uwp/api/windows.graphics.imaging.bitmapdecoder.getpixeldataasync#Windows_Graphics_Imaging_BitmapDecoder_GetPixelDataAsync
      var data = await dec.GetPixelDataAsync();

      imgBytes = data.DetachPixelData();
      imgWidth = dec.OrientedPixelWidth;
      imgHeight = dec.OrientedPixelHeight;

      // FYI, Overall image L-R, Top-Bottom.
      // Groups in reverse order: BGRA BGRA BGRA

      ByteArrayToPixelMap(imgBytes);
    }
  }
}

Dynamically Populate a UWP Grid using C#

This technique works for columns and rows, you just have to use the correct definition object, ColumnDefinition or RowDefinition.

Note the references to your specific grid instance (e.g. myGrid) and Grid methods (e.g. Grid.SetRow()).

myGrid.Children.Clear();
myGrid.ColumnDefinitions.Clear();

ColumnDefinition colDef;

for(int i = 0; i < myArray.Length; i++)
{
  colDef = new ColumnDefinition();
  myGrid.ColumnDefinitions.Add(colDef);
  colDef.MinWidth = 50;

  // Create a new instance of a TextBlock
  // or any other control; making a factory 
  // method(s) is usually wise.
  TextBlock txt = MyTextBlockFactoryMethod();
  myGrid.Children.Add(txt);
  Grid.SetRow(txt, 0);
  Grid.SetColumn(txt, i);
  txt.Text = myArray[i].ToString();
}

Application Version Info Panel

Here’s how to build a simple panel that shows the user the version info of your UWP application.

First the XAML / UI:

<Border x:Name="pnlInstallLocation" Padding="10" CornerRadius="5" Background="#FFB9D2F6" Margin="0,5,15,0" RelativePanel.AlignLeftWithPanel="True" RelativePanel.AlignRightWithPanel="True">
  <RelativePanel>
    <TextBlock x:Name="lbllocation">Install Details</TextBlock>
    <TextBlock x:Name="txtBuildDetails" Text="[build details]" RelativePanel.Below="lbllocation" RelativePanel.AlignRightWithPanel="True" RelativePanel.AlignLeftWithPanel="True" FontSize="12" Padding="2" MinHeight="12" Margin="10,0,0,0" />
    <TextBox x:Name="txtVersion" Text="[version]" RelativePanel.Below="txtBuildDetails" RelativePanel.AlignRightWithPanel="True" RelativePanel.AlignLeftWithPanel="True" FontSize="12" Padding="2" IsReadOnly="True" BorderThickness="0" Background="Transparent" MinHeight="12" Margin="10,0,0,0" />
    <TextBox x:Name="txtLocation_App" Text="[location]" RelativePanel.Below="txtVersion" RelativePanel.AlignRightWithPanel="True" RelativePanel.AlignLeftWithPanel="True" FontSize="12" Padding="2" IsReadOnly="True" BorderThickness="0" Background="Transparent" Height="20" MinHeight="12" Margin="10,0,0,0" />
    <TextBox x:Name="txtLocation_DB" Text="[location]" RelativePanel.Below="txtLocation_App" RelativePanel.AlignRightWithPanel="True" RelativePanel.AlignLeftWithPanel="True" FontSize="12" Padding="2" IsReadOnly="True" BorderThickness="0" Background="Transparent" Height="20" MinHeight="12" Margin="10,0,0,0" />
  </RelativePanel>
</Border>

The code behind:

using Windows.ApplicationModel;

private void Page_Loaded(object sender, RoutedEventArgs e)
{
  Package package = Package.Current;
  PackageId packageId = package.Id;
  PackageVersion version = packageId.Version;

  txtBuildDetails.Text = string.Format("{3} ({4}){0}Package {2}{0}Installed {1}", 
    Environment.NewLine,
    package.InstalledDate, package.Id.Name,
    package.DisplayName, package.Id.Architecture);

  txtVersion.Text = string.Format("Version {0}.{1}.{2}.{3}", version.Major, version.Minor, version.Build, version.Revision);

  txtLocation_App.Text = string.Format("Application folder {0}", package.InstalledLocation.Path);
  txtLocation_DB.Text = string.Format("Database folder {0}", ApplicationData.Current.LocalFolder.Path);


  #if DEBUG
  txtVersion.Foreground = new SolidColorBrush(Windows.UI.Colors.DarkRed);
  #endif
}

 

This code has been lifted from a working app, I haven’t done a detailed checking of the code for this post, but everything needed should be present – uses Windows 10 Anniversary Edition (10.0; Build 14393).

Using Symbols in UWP UI Controls

To use a symbol character from a font like Segoe MDL2 Assets or Segoe UI Symbol as the visible content / text of a UWP control in XAML, use the following code, where 1234 is the correct symbol code.

E.g.:

Content="&#xE15E;"

To set in C# use:

lblMySymbol.Content="\u1234;"

Also see:

  1. https://docs.microsoft.com/en-us/windows/uwp/design/style/segoe-ui-symbol-font
  2. https://www.google.co.nz/search?q=segoe+mdl2+assets+cheatsheet
  3. http://www.kreativekorp.com/charset/font.php?font=Segoe+UI+Symbol

About These Code Posts

I no longer code to earn a living, I code for pleasure.  The code snippets posted here are mainly for my own reference – and to benefit anyone who can use them.