Tag Archives: Artificial Intelligence

A Human-Level Tactical Artificial Intelligence at Ligny

Map 159 from the superb, “West Point Atlas of the Napoleonic Wars,” (Esposito & Elting, 1999, Stackpole). Scanned from the author’s collection. Click to enlarge.

The seeds of Napoleon’s defeat at Waterloo were sown two days earlier at his victory near Ligny. Napoleon needed to surround and completely remove the Prussian army as a viable force on the battlefield. Instead, they escaped to Wavre in the north and resurfaced at the worst possible time on Napoleon’s right flank two days later at Waterloo.

MATE1)Machine Analysis of Tactical Environments 2.0 is now capable of analyzing the battle of Ligny, June 16, 1815 from both the Blue (L’Armée du Nord on the offensive) and Red (Prussian on the defensive) positions. (MATE is the AI behind General Staff: Black Powder. For more information about MATE see these links).

The Ligny map was donated by Glenn Frank Drover. Jared Blando is the artist. Ed Kuhrt did the elevation, roads and mud terrain overlays. The unit positions are from the West Point Atlas of the Napoleonic Wars (above) and from David Chandler’s Waterloo: The Hundred Days. If any one has a better source for unit positions, please contact me directly.

Screen capture of the Ligny scenario in General Staff. Elevation and slope layers enhanced.

Below is MATE’s analysis from the Blue (L’Armée du Nord) perspective. MATE correctly identifies the key positions and realities of the battlefield:

  • Red is on the defensive
  • Red has an exposed flank
  • There are two key choke points on the route to Red’s exposed flank

MATE then creates an appropriate Course of Action (COA) for Blue:

  • Battle Group #1 (The French III Corps) is assigned the flanking maneuver.
  • Battle Group #0 (The Imperial Guard) is assigned the objective of St. Amand with the support of Battle Group #4 (IV Corps cavalry and reserve artillery).
  • Battle Group #2 (IV Corps) demonstrates against Ligny.
  • Battle Group #3 (The Cavalry Reserve) seizes Balatre and a crucial bridge located there.

MATE’s analysis of Blue’s position at Ligny. Screen capture. Click to enlarge.

A log of MATE’s thought processes, with my commentary, follows:

Text output of MATE’s analysis of Blue’s position at Ligny. Click to enlarge.

MATE also analyzed Ligny from the Prussian (Red) perspective:

Screen capture of MATE’s analysis of Ligny for Red (Prussian army). MATE recognizes the two choke points on the route of the enemy’s flank attack and dispatches cavalry units to cover these critical areas. Click to enlarge.

MATE, analyzing the Prussian (Red) position correctly recognizes that it is on the defensive, it has an exposed flank, there are two crucial choke points on the route that Blue will take on its flanking maneuver and dispatches two cavalry units to cover the bridges. A log of MATE’s thought processes, with my commentary, follows:

Text output of MATE’s analysis of Red’s position at Ligny. Click to enlarge.

Critique of MATE’s analysis:

As the author of MATE any critique I have of its performance should be taken with a grain of salt (also, see this video). If I was back in academia I would put together twenty or thirty Subject Matter Experts (SMEs), set up a double blind web site, get all the SME’s solutions to the problem, and compare their solutions to MATE’s. If they match to a statistical significance it proves the ‘human-level’ part. But, I’m not in academia anymore and you’ll just have to take my word for it. That said, MATE did what I expected it to do.

It first sussed out if it was on offense or defense and what it had to do to win.

Then, as Blue, MATE discovered a back door to Red’s position and ordered a classic enveloping maneuver. MATE assigned Blue Battle Group #1 the task of implementing the flanking maneuver. Blue Battle Groups #0, #4 and #2 are the fixing force. See my paper, Implementing the Five Canonical Offensive Maneuvers in a CGF Environment (free download here) for details and algorithms.  The Blue Cavalry Reserve is given the COA to seize the town of Balatre. This, in my opinion, is a pretty good tactical plan.

When MATE finds itself on defense, as it does as Red at Ligny, one of the first things it does is ask itself, “how would I attack myself?” So, of course it finds the back door right away. Then it compiles a list of available units that are not actively engaged in holding crucial parts of a defensive line, selects the optimal (fastest) units and assigns them orders to defend the crucial choke points. This was a better plan than Field Marshal Gebhard Leberecht von Blücher had. So, again, I’m going to argue that MATE is operating at a ‘human-level’.

As always, please feel free to write me with the questions or comments. MATE is going to take a look at Antietam next.

References

References
1 Machine Analysis of Tactical Environments

A Human-Level Tactical Artificial Intelligence at Bull Run

West Point Atlas Map 20; the situation near Manassas & Centreville, July 20, 1861. Click to enlarge. From the Library of Congress; original source.

I‘ve been looking for interesting tactical problems for MATE1)Machine Analysis of Tactical Environments 2.0 to solve and I found a good one after reading William C. Davis’s Battle at Bull Run.2)1995, Stackpole Books,  Mechanicsburg, PA The actual battle (called 1st Bull Run by the Union who named battles after nearby waterways and 1st Manassas by the Confederates who labeled battles from nearby geographic features and cities) was a tragicomedy fought on July 21, 1861 in which both commanders (Irvin McDowell for the Union and P. G. T. Beauregard for the Confederates) had little control of their own forces after their initial battle orders were given. Indeed, the battle came down to a series of charges up and down Henry Hill with units committed piecemeal as they arrived on the field. Large elements of both armies were never committed. All in all, not a particularly interesting tactical situation for MATE to analyze.

However, the tactical position the day before (see West Point Atlas map #20, above) is quite a different situation. The Union army is massed at Centreville (Washington, D. C. is off the map, about 30 miles to the east). The two armies are separated by Bull Run which can only be crossed at eleven fords and bridges. Confederate general Beauregard is certain that McDowell’s attack will be almost due south from Centreville and will cross Bull Run at Mitchell’s and Blackburn’s fords. He has assembled almost all of his forces there. This is a tactical situation of which avenues of attack are open and which are closed.

Troop positions and topographical data fed to MATE for this analysis come from the McDowell Map, below:

Map of the battlefield of Bull Run, Virginia. Brig. Gen. Irvin McDowell commanding the U.S. forces, Gen. [P.] G. T. Beauregard commanding the Confederate forces, July 21st 1861 from the Library of Congress. Click to enlarge.

MATE’s assessment of this situation from the Confederate (RED) perspective is below. I use a program called the AI Editor (which, ironically, doesn’t actually edit AI) to observe what MATE is thinking and seeing.

Screen shot of the AI Editor. Click to enlarge.

The left window contains a series of predicate statements, conclusions and inferences. Predicate statements to MATE are simple factual statements that MATE knows to be true; e. g. statement #4: The enemy needs 300 Victory Points to win is a basic factual statement. MATE can combine statements (such as #4 and #5: The enemy currently controls 125 Victory Points) to come to a logical conclusion (indicated by beginning the new statement with logical symbol ”∴” or therefore): #6∴ The enemy needs to seize 175 more Victory Points.

The left window is divided into two windows of scrolling text. I printed out the complete list of all statements and conclusions and added commentary (below) so you can follow the thread of MATE’s thought processes:

List of premises and conclusions with commentary from MATE’s analysis of Manassas. Click to enlarge.

The right window shows the graphic output of some of MATE’s views of the battlefield (see Layers: Why a Military Simulation Is Like a Parfait). In the above screen shot it is displaying the terrain and elevation layers of the map plus all RED and BLUE forces. The yellow line is how MATE would attack if it were BLUE. Yes, that is correct. MATE analyzes its own defensive position by planning to attack it from the enemy’s perspective. The yellow line (the path using the road net) is how it would turn its own flank. It was this analysis that triggered the creation of statement #31: I have an exposed flank! To see the complete algorithm click here (PDF). The Red line is the optimal route of the 30th Virginia Cavalry to Sudley Ford indicated on the screen by the black box labeled CHOKE POINT.

MATE’s analysis of Manassas certainly appears to be a reasonable analysis and solution to this tactical problem.  It also generated a COA (Course of Action) ordering a regiment of cavalry to secure a critical choke point. This, in fact, was better than Confederate General Beauregard’s actual performance.

Is there more work to do? Certainly. MATE uses heuristics. Here is the classic definition of heuristics: “A heuristic function, also simply called a heuristic, is a function that ranks alternatives in search algorithms at each branching step based on available information to decide which branch to follow.”

Here is my definition of a heuristic: a function that groks3)to understand profoundly and intuitively from Heinlein’s Stranger in a Strange Land the problem. MATE uses dozens of heuristic algorithms. MATE is pretty good at discovering – and pouncing – on an exposed flank. MATE groks exposed flanks. MATE also groks interior lines, the high ground, the road net, and constricted avenues of attack and retreat. That may not be a long list but it ticks more boxes than most 19th century generals.

References

References
1 Machine Analysis of Tactical Environments 2.0
2 1995, Stackpole Books,  Mechanicsburg, PA
3 to understand profoundly and intuitively from Heinlein’s Stranger in a Strange Land

General Staff will Ship in 2022

A few of Grant’s unsuccessful attempts to take Vicksburg. Map from West Point Atlas of American Wars. Library of Congress. Click to enlarge.

Grant took Vicksburg on his sixth attempt. He first tried a direct movement, east of the Mississippi, against the city but was forced to turn back after Confederate General Van Dorn’s raid destroyed his supply base. Sherman next attempted a direct assault and failed at Champion’s Bluff. Grant then tried to cut Vicksburg off from the Mississippi by digging a great canal across the neck of a giant oxbow. Unusually low water levels in the river that year caused another failure. Grant also tried more canal digging and river clearing at Lake Providence and the Yazoo Pass without luck.

Col. Theodore Lyman, a staff officer of the Army of the Potomac, wrote of Grant, “He habitually wears an expression as if he had determined to drive his head through a brick wall, and was about to do it. I have much confidence in him.”

Grant finally succeeded with his brilliant amphibious landing south of Vicksburg followed by a series of ‘must win’ battles that concluded with besieging the city on May 22, 1863. Confederate General Pemberton surrendered Vicksburg on July 4, 1863.

The General Staff: Black Powder Wargaming System is comprised of four separate programs: the Army Editor, Map Editor, Scenario Editor and the actual Game Engine, itself. The first three have been done for some time (it’s what I use to produce scenarios for testing the AI). They were written in C# using Microsoft’s Windows Presentation Foundation (WPF) which is often used for creating desktop business applications. This decision made sense because the Army, Map and Scenario Editors function not unlike a suite of business applications that create and share data with a common look and feel. However, while WPF was a good development decision for the Editor programs it turned out not to be a great tool for creating the Game Engine.

A flow chart of the General Staff: Black Powder Wargaming System. Files created in the Army Editor and the Map Editor are combined in the Scenario Editor to create a scenario or simulation. Click to enlarge.

What the Game Engine does is it takes data created in the three Editor programs, displays them in a visually exciting way, enables the user to make input (orders), and simulates those orders (as well as the AI’s orders) and the resulting combat to the highest degree of accuracy possible. That sounds like a lot, but it really isn’t. And the heavy lifting that the Game Engine is being asked to do is all visual: making everything look good on the screen.

Recently, I’ve been looking at Grand Tactician: Civil War which is an immersive voyage into 19th century American typography, cartography, tactics and grand strategy. I eagerly look forward to having the time to do more than gaze upon it in amazement 1)Full disclosure, the developers were kind enough to give me a review copy of Grand Tactician and, damn, it looks fantastic.. I reached out to Ilja on their development team and he convinced me that we should complete the General Staff Game Engine in Unity (which is what they used for Grand Tactician). What they accomplished visually with Grand Tactician was precisely the look and feel that I wanted for General Staff 2)I’m talking about the typography and the UI, not thousands of tiny 3D soldiers on the battlefield; which look really great, but, General Staff is not that kind of a wargame..

Consequently, we are officially announcing that we are abandoning writing the General Staff: Black Powder Game Engine in WPF and, instead, will be creating it using the Unity engine. We believe that this will produce a far superior visual experience 3)Please note: we will not being using Unity to create a 3D simulation with tiny soldiers. I like playing those kind of games, but that’s not what General Staff is..

Because Unity is designed to facilitate the creation of computer games, this move will greatly accelerate the completion of the General Staff Game Engine and will allow us to ship General Staff: Black Powder this year.

I am certainly not U. S. Grant, though I like to think I have some of that bulldog determination. Like Grant before Vicksburg, I’ve had to acknowledge that our first attempt will not succeed and we’re moving on to another plan. I am sorry for the delay in the release of General Staff and am confident that, with Unity, we will soon be reporting some good news.

I am also very excited to announce that my good friend, Damien, will be joining the team to specifically work in Unity completing the General Staff Game Engine. Damien lives in Ireland and has been a devout wargamer since playing Arnhem on the ZX Spectrum. When not enjoying a Guinness or a Jamesons, he’ll be playing tabletop or computer games, whether that means pushing cardboard counters across hex boards, drawing LOS between badly painted miniatures on the tabletop, or day-long sessions in front of a computer – where he particularly enjoys playing against a challenging, non-cheating AI. Damien is a 28 year veteran of the IT industry, having worked on everything from telecoms, healthcare and financial systems, across Mainframes, Linux, Unix and Windows, and co-inventor of two messaging system patents. He has a particular passion for language processing and domain specific languages, originating from his interest in the elegance of recursive descent compiler construction.

References

References
1 Full disclosure, the developers were kind enough to give me a review copy of Grand Tactician and, damn, it looks fantastic.
2 I’m talking about the typography and the UI, not thousands of tiny 3D soldiers on the battlefield; which look really great, but, General Staff is not that kind of a wargame.
3 Please note: we will not being using Unity to create a 3D simulation with tiny soldiers. I like playing those kind of games, but that’s not what General Staff is.

Why Machines May Kill Us In Our Sleep

An amazing screen capture of the AI’s solution to a problem. It has found a 1 pixel gap between the data and the edge of the screen and is exploiting it to successfully find an ‘open flank’ of Red. Click to enlarge.

Professor Alberto M. Segre was my thesis advisor and one day he said to me, “You know when your AI is really working because it will surprise you.” Today I got to have one of those weird surprises.

The screen shot (above) is a visual representation of what the AI is up to. You won’t get to see this in the actual game. The program that’s running is called the AI Editor which is a bit of a misnomer because you don’t actually edit the AI in it; you mostly just get to observe what it’s doing. There’s a lot of stuff going on in the above image. There are multiple layers visually displaying different types of data (check out the blog – Layers: Why a Military Simulation is Like a Parfaitfor more information about these). But, what interests us are the AI layers: Battle Groups, Objectives, and that thin yellow line that snakes from a group of blue units, crossing Antietam Creek at the Middle Bridge and then, amazingly, exploiting a data anomaly to reach its goal: a point far behind enemy lines.

Some background on the situation:

The map of the Antietam Battlefield (screen shot) with terrain and elevation layers displayed. Click to enlarge.

Underlying all the clutter from the first screen capture, top, is the battle of Antietam (above). The map has been rotated 90 degrees to the left so north is now pointing to the left; east is at the top of the screen.

After adding Blue (Union) and Red (Confederate) units to the map in their historical positions at 0600 September 17, 1862 the AI performed a tactical analysis from the perspective of Blue.

The AI ‘strategic’ analysis for Antietam playing Blue (Union).

The above are a list of Predicate Statements all of which the AI knows to be true. Statements preceded by the logical sign ∴ (therefore) are conclusions, or inferences, derived from the predicate statement referenced in the brackets. It is this analysis that determines if the AI will be on the offensive or defensive and what its objectives will be.

Next, the AI performs Range of Influence (ROI) calculations for the entire observable battlefield. I plan on doing a video about this later, but for now the darker the red (in the topmost screen capture) the more – and more powerful – weapons the Red army can bear on that point.  The AI next divides all the units on the map into a forest of minimum spanning trees called Battle Groups. I want to do a video about this, too. However, if you can’t wait, these subjects are covered in my paper, Implementing the Five Canonical Offensive Maneuvers in a CGF Environment (free download).

Again, referring to the top screenshot you can see the AI’s calculations to this point:

  • It has determined it (Blue) will be on the offensive.
  • It has calculated enemy ROI.
  • It has assigned objectives to the first Battle Group.

Flanking Algorithm published in, “Algorithms for Generating Attribute Values for the Classification of Tactical Situations”. Click to enlarge.

Now the AI needs to determine if the enemy has an ‘open or unanchored flank’. In Algorithms for Generating Attribute Values for the Classification of Tactical Situations I published the Algorithm for Flanking Attribute Value Function (right). It basically comes down to this: can the AI trace an unbroken path from the center of the Blue Battle Group to a specific point (called the Retreat Point) far behind enemy lines without crossing into ‘No Go Areas’ (water, swamp) or entering any area controlled by Red’s ROI (literally the red areas in the topmost screen shot).

The reason that I was using Antietam as a test case for anchored / unanchored flanks is because years ago I had analyzed the battle for my doctoral thesis and knew it to be a classic example of anchored flanks; Lee’s left flank rests on the Potomac and his right flank is anchored on the Antietam. Granted, the Confederate flanks were held by Stuart’s cavalry with a little horse artillery support but they were still, by definition, anchored flanks.

Due to an error in the data that made up the Antietam terrain map a 1 pixel (about 3.8 meter wide) strip of ‘no terrain’ was inserted at the far right hand edge of the map (see blow up of screen capture, right; it’s the thin line between the water, represented in red, and the brown edge of the map). This meant there was a ‘land bridge’ across Antietam Creek where none existed in real life. A digital parting of the Red Sea, if you will. But, by the rules of the game the AI perfectly performed its function. There was no error in the AI – again, the AI performed better than I had dared hope – the error was with the data set.

And that’s how fifty years from now I can see a cyber-detective standing over the chalk marks around a body saying, “Yeah, the machine performed perfectly, brilliantly, in fact. But, the error in the data set killed him.”

It’s already happened in real life. For cars with autopilot the data set of the world in which it operates is crucial. However, “against a bright spring sky, the car’s sensors system failed to distinguish a large white 18-wheel truck and trailer crossing the highway, Tesla said. The car attempted to drive full speed under the trailer, “with the bottom of the trailer impacting the windshield of the Model S”, Tesla said.” The driver died. The AI functioned perfectly. But, the error in the data set killed him.

So, I fixed the error in the data set (probably caused by not using the right values in InkScape when I converted the Antietam Water.bmp into paths) and imported it back into the Antietam map using the General Staff Map Editor, saved it out, and ran the AI Editor again and saw this:

The AI did not display a yellow path from the center of the Blue Battle Group to the Red Retreat Point because none existed. Instead, it just wrote the first Predicate Statement in the Tactical Analysis stack: “Red’s flanks are anchored”.

Again, the machine was performing perfectly. And its results were no longer surprising.

Addendum

I recently got to experience this again (though this time it was caused by a different data bug) when I was reviewing the AI’s decisions at the battle of Manassas:

Because the Range of Influence was not calculating the very bottom row the AI found another, perfectly legal, way to reach its goal. Screen shot from the General Staff AI Editor. Click to enlarge.

In this instance, the error in the database was caused by the Range of Influence (basically a map of what red and blue can see and hit) not calculating the very last row. Consequently, the AI was able to legally trace a path from the blue forces in the northeast to their goal at the bottom of the map.

After this bug was corrected the AI performed as expected:

The AI correctly sees going around red’s left flank as the solution to the problem. Screen shot from General Staff AI Engine. Click to enlarge

In the above screen shot the AI has demonstrated the correct solution to the tactical problem facing blue at Manassas on July 20, 1861 (the day before the actual battle). Red’s left flank is unanchored. It’s wide open. Note how the AI identifies the one choke point (Sudley Springs Ford) in the plan.

So, the AI surprised me again. I think it’s looking pretty good. When you play against it, watch your flanks.

Feeding the Machine

The famous Turing Machine1)It was first described in Turing’s, “On Computing Machines with an Application to the Entscheidungsproblem,” in 1937 which can be downloaded here: https://www.cs.virginia.edu/~robins/Turing_Paper_1936.pdf. Also a very good book on the subject is Charles Petzold’s, “The Annotated Turing: A Guided Tour through Alan Turing’s Historic Paper on Computability and the Turing Machine.” was a thought experiment and, until recently did not physically exist 2)Yes, somebody has built one and you can see what Turing described here: https://www.youtube.com/watch?v=E3keLeMwfHY . When computer scientists talk about machines we don’t mean the, “lumps of silicon that we use to heat our offices,” (thanks Mike Morton for this wonderful quote), but, rather, we mean the software programs that actually do the computing. When we talk about Machine Learning we don’t think that the physical hardware actually learns anything. This is because, as Alan Turing demonstrated in the above paper, the software functions as a virtual machine; albeit, much more efficiently than creating a contraption with pens, gears, rotors and an infinitely long paper strip.

When I talk about, “feeding the machine,” I mean giving the program (the AI for General Staff is called MATE: Machine Analysis of Tactical Environments and the initial research was funded by DARPA) more data to learn from. Yesterday, the subject at machine learning school was Quatre Bras.

Screen shot of the General Staff AI Editor after analysis of Quatre Bras and calculating the flanking Schwerpunkt or point of attack (blue square).  Click to enlarge.

The MATE tactical AI algorithms produce a plan of attack around a geographic point on the battlefield that has been calculated and tagged as the Schwerpunkt, or point where maximum effort is to be applied. In the above (Quatre Bras) scenario the point of attack is the extreme left flank of the Anglo-Allied (Red) army. I apply the ‘reasonableness test’ 3)Thank you Dennis Beranek for introducing me to the concept of ‘reasonableness test’. See https://www.general-staff.com/schwerpunkt/ for explanation and think, “Yes, this looks like a very reasonable plan of attack – a flanking maneuver on the opponent’s unanchored left flank – and, in fact, is a better plan than what Marhshal Ney actually executed.

It would be good at this point to step back and talk about the differences in ‘supervised’ and ‘unsupervised’ machine learning and how they work.

Supervised machine learning employs training methods. A classic example of supervised learning is the Netflix (or any other TV app’s) movie recommendations. You’re the trainer. Every time you pick a movie you train the system to your likes and dislikes. I don’t know if Netflix’s, or any of the others, use a weighting for how long (what percentage watched over total length of show) watched but that would be a good metric to add in, too. Anyway, that’s how those suggestions get flashed up on the screen: “Because you watched Das Boot you’ll love The Sound of Music!”  Well, yeah, they both got swastikas in them, so… 4)Part of the problem with Netflix’s system is that they hire out of work scriptwriters to tag each movie with a number of descriptive phrases. Correctly categorizing movies is more complex than this.

Supervised machine learning uses templates and reinforcement. The more the user picks this thing the more the user gets this thing. MATE is unsupervised machine learning. It doesn’t care how often a user does something, it cares about always making an optimal decision within an environment that it can compare to previously observed situations. Furthermore, MATE is a series of algorithms that I wrote and that I adjust after seeing how they react to new scenarios. For example, in the above Quatre Bras scenario, MATE originally suggested an attack on Red’s right-flank. This recommendation was probably influenced by the isolated Red infantry unit (1st Netherlands Brigade) in the Bois de Bossu woods.  After seeing this I added a series of hierarchical priorities with, “a flank attack in a woods (or swamp) is not as optimal as an attack on an exposed flank with clear terrain,” as a higher importance than pouncing on an isolated unit.  And so I, the designer, learn and MATE learns.

My main concern is that MATE must be able to ‘take care of itself’ out there, ‘in the wild’, and make optimal decisions when presented with previously unseen tactical situations. This is not writing an AI for a specific battle. This is a general purpose AI and it is much more difficult to write than a battle specific AI. One of the key aspects of the General Staff Wargaming System is that users can create new armies, maps and scenarios. MATE must make good decisions in unusual circumstances.

Previously, I have shown MATE’s analysis of 1st Bull Run and Antietam. Below is the battle of Little Bighorn in the General Staff AI Editor:

The battle of Little Bighorn in the General Staff AI Editor. Normally the MATE AI would decline to attack. However, when ordered to attack, this is MATE’s optimal plan. Click to enlarge.

I would like to expose MATE to at least thirty different tactical situations before releasing the General Staff Wargame. This is a slow process. Thanks to Glenn Frank Drover of Forbidden Games, Inc. for donating the superb Quatre Bras map. He also gave us maps for Ligny and Waterloo which will be the next two scenarios submitted to MATE. We still have a way to go to get up to thirty. If anybody is interested in helping to create more scenarios please contact me directly.

References

References
1 It was first described in Turing’s, “On Computing Machines with an Application to the Entscheidungsproblem,” in 1937 which can be downloaded here: https://www.cs.virginia.edu/~robins/Turing_Paper_1936.pdf. Also a very good book on the subject is Charles Petzold’s, “The Annotated Turing: A Guided Tour through Alan Turing’s Historic Paper on Computability and the Turing Machine.”
2 Yes, somebody has built one and you can see what Turing described here: https://www.youtube.com/watch?v=E3keLeMwfHY
3 Thank you Dennis Beranek for introducing me to the concept of ‘reasonableness test’. See https://www.general-staff.com/schwerpunkt/ for explanation
4 Part of the problem with Netflix’s system is that they hire out of work scriptwriters to tag each movie with a number of descriptive phrases. Correctly categorizing movies is more complex than this.