Smarter AI with NavMesh: Patrol, Chase, and React

NavMesh-based AI: Don’t just set it and forget it. Far too often, I see developers slap a NavMeshAgent onto an enemy, point it in a direction, and call it a day. The result? Predictable, unengaging gameplay. Let’s ditch that mediocrity and build an AI that actually reacts to the player, using a robust patrol system, a proper chase mechanic, and a reliable return-to-patrol behavior.

Baking Your Battlefield: NavMesh Basics

The foundation of our intelligent enemy lies in the NavMesh. This is a crucial step; a poorly baked NavMesh leads to janky movement and AI that gets stuck on every pebble.

1. Static is Key: Ensure all static environment elements (floors, walls, obstacles) are marked as “Static” in the Inspector. This tells Unity to include them in the NavMesh generation.
2. Navigation Static: Even better, use the Navigation Static checkbox directly within the static flags, which allows finer-grained control.
3. Bake it Till it’s Golden: Navigate to Window > AI > Navigation. In the “Bake” tab, adjust the settings. The Agent Radius and Agent Height are particularly important; they dictate how closely the AI can move to obstacles. Too small, and the AI will get stuck. Too large, and they won’t be able to navigate tight corridors. Experiment to find the sweet spot.
4. Pitfall: A common mistake is neglecting the Max Slope setting. If your environment has ramps or slopes steeper than this value, the AI will simply refuse to climb them.

Patrol Like a Pro: Waypoint System

Forget simple random wandering. A well-defined patrol route gives your enemy a sense of purpose and predictability (in a good way!).

1. Waypoint Creation: Create empty GameObjects to serve as your waypoints. Position these strategically around your level. Name them sequentially (e.g., Waypoint01, Waypoint02, etc.)
2. The Patrol Script: Create a new C# script called PatrolRoute and attach it to your enemy. This script will manage the waypoint navigation.

using UnityEngine;
using UnityEngine.AI;
using System.Collections;

public class PatrolRoute : MonoBehaviour
{
    public Transform[] waypoints; // Array of waypoint transforms
    public float patrolSpeed = 3.0f;
    public float waypointReachedDistance = 1.0f;

    private NavMeshAgent agent;
    private int currentWaypointIndex = 0;

    void Start()
    {
        agent = GetComponent<NavMeshAgent>();
        agent.speed = patrolSpeed;

        if (waypoints.Length == 0)
        {
            Debug.LogError("No waypoints assigned to PatrolRoute script on " + gameObject.name);
            enabled = false; // Disable the script if no waypoints are assigned
            return;
        }

        SetDestination();
    }

    void Update()
    {
        if (agent.remainingDistance <= waypointReachedDistance && !agent.pathPending)
        {
            currentWaypointIndex = (currentWaypointIndex + 1) % waypoints.Length;
            SetDestination();
        }
    }

    void SetDestination()
    {
        agent.destination = waypoints[currentWaypointIndex].position;
    }
}

3. Array Assignment: In the Inspector, drag your waypoint GameObjects into the waypoints array.
4. Fine-Tuning: Adjust the patrolSpeed and waypointReachedDistance variables to suit your game’s pacing and level design.

Challenge: Many beginners hardcode waypoint indices. This makes it difficult to add or remove waypoints later. The modulo operator (%) in the currentWaypointIndex = (currentWaypointIndex + 1) % waypoints.Length; line ensures that the AI loops back to the beginning of the patrol route seamlessly.

Chase and Return: A Matter of Priorities

The real magic happens when the AI reacts to the player. The key is to prioritize chasing when the player is in range, and seamlessly return to the patrol route when they’re not.

1. The Chase Script: Create a new C# script called ChaseBehavior and attach it to your enemy.

using UnityEngine;
using UnityEngine.AI;

public class ChaseBehavior : MonoBehaviour
{
    public float chaseDistance = 10.0f;
    public float chaseSpeed = 7.0f;
    public float returnToPatrolDistance = 15.0f;
    public Transform player;

    private NavMeshAgent agent;
    private PatrolRoute patrolRoute;
    private bool isChasing = false;
    private Vector3 lastPatrolDestination;

    void Start()
    {
        agent = GetComponent<NavMeshAgent>();
        patrolRoute = GetComponent<PatrolRoute>();

        if (player == null)
        {
            player = GameObject.FindGameObjectWithTag("Player").transform;
            if (player == null)
            {
                Debug.LogError("No Player found.  Ensure player has the 'Player' tag.");
                enabled = false;
                return;
            }
        }
    }

    void Update()
    {
        float distanceToPlayer = Vector3.Distance(transform.position, player.position);

        if (distanceToPlayer <= chaseDistance)
        {
            if (!isChasing)
            {
                isChasing = true;
                lastPatrolDestination = agent.destination;
                patrolRoute.enabled = false; // Disable patrol while chasing
                agent.speed = chaseSpeed;
            }
            agent.destination = player.position; // Chase the player
        }
        else if (isChasing && distanceToPlayer > returnToPatrolDistance)
        {
            isChasing = false;
            patrolRoute.enabled = true; // Re-enable patrol
            agent.speed = patrolRoute.patrolSpeed;
            //Option 1: Resume last route
            agent.destination = lastPatrolDestination;
            //Option 2: Return to closest patrol point
            //float minDistance = Mathf.Infinity;
            //int closestWaypointIndex = 0;
            //for (int i = 0; i < patrolRoute.waypoints.Length; i++)
            //{
            //    float distanceToWaypoint = Vector3.Distance(transform.position, patrolRoute.waypoints[i].position);
            //    if(distanceToWaypoint < minDistance)
            //    {
            //        minDistance = distanceToWaypoint;
            //        closestWaypointIndex = i;
            //    }
            //}
            //patrolRoute.currentWaypointIndex = closestWaypointIndex;
            //patrolRoute.SetDestination();

        }
    }
}

2. Variable Assignment: In the Inspector, assign the Player GameObject to the player variable. Adjust chaseDistance, chaseSpeed, and returnToPatrolDistance to fine-tune the AI’s behavior.
3. Don’t Forget the Tag: Make sure your Player object is tagged as “Player” in the inspector.

Pitfall: Ensure the NavMeshAgent component is attached to the same GameObject as the PatrolRoute and ChaseBehavior scripts.

From Zero to Hero: Avoiding Common AI Mistakes

Many developers struggle with AI that feels “dumb.” Here’s how to avoid those pitfalls:

• Obstacle Avoidance: Make sure the NavMeshAgent's Obstacle Avoidance Priority is set correctly. Lower priority agents will yield to higher priority ones, preventing them from getting stuck in doorways.
• Stopping Distance: The stoppingDistance property of the NavMeshAgent dictates how close the AI needs to get to its target before stopping. Adjust this to prevent jittering or overshooting.
• Animation Synchronization: Don’t forget to synchronize your AI’s animations with its movement. Use the NavMeshAgent.velocity to drive your Animator parameters.

By following these steps, you’ll have a solid foundation for creating engaging and reactive AI in your Unity games. Forget the boring, predictable enemies – let’s build AI that actually challenges the player!