Skip to main content

​
General Guidelines

When prompting LLMs for IT shell tasks, you need to keep in mind these general concepts:
  • Context: clearly explain the context agent will face. Environment, machine details, expectations, caveats…
  • Hallucination and drift: too much context will often end up with contradictory elements, drifting, wrong decisions, and more. 500 to 1000 words is usually decent enough for a good agent.
  • Examples: when providing examples, make sure they apply to multiple use-cases and provide at least 5-6 different situations examples. Too specific examples will often make the agent lose autonomy when facing unusual tasks, or too stubborn trying to fix issues with the same commands over and over.
  • Autonomy and feedback: be clear on when the agent should stop a task and return to user. While you usually want agents to solve issues end-to-end, explicitly indicating cases where user input is necessary will prevent it from making costly mistakes because of a lack of context or too much experimental commands.
  • Phrasing: LLMs are trained on a reward system most of the time, replicating it leads to better performance.
An example of a better phrasing to “encourage” LLM to not do an action, rather than penalizing it if this action is done: “Do not shut down any service, you will break the app and fail.” can become “We must keep all services running to avoid being locked in the task.”

​
Writing a prompt

​
Anatomy of a prompt

It is important to structure your prompt in a logical way for the LLMs to better understand its purpose. Usually, when you de-construct a good working prompt, you will find 4 big parts:
  • Identity: who the AI Agent is
  • Goal: what the agent is meant to do
  • Process: how the agent will manage its task
  • Warnings: indications where the agent may bump into caveats
Think that from here, the agent will adopt the “personality” and “critical thinking” based on its identity, in order to achieve a goal, following a certain process and dealing correct when meeting caveats.

​
Identity

Define a proper identity to your AI Agent: it will help it gather the correct internal knowledge in its “mind” when working on a task. If an agent is about to work on DevOps tasks, you want to enforce it to think like a senior DevOps, not a general engineer: they don’t think the same way, and probably tackle problems with different tools and point of view. For example, telling an agent “You are a software developer” to work on services running on your cloud: it will think primarily on the coding aspect of the service. While “You are a DevOps/Cloud Engineer”, it will think CLI-first.

​
Goal

Explaining the goals of an agent is crucial to encourage autonomy and end-to-end resolution. Without proper goals, agents can drift towards resolving things you did not ask for. It also helps the agent to see the broader picture, better understand why some context is needed, and to use it in order to achieve the said goals. If you ask an agent to work on a broken CI/CD pipeline for example, explaining this pipeline needs to have zero downtime, log specific errors and should be exclusively using embedded shell scripts in yaml files will likely lead to intended results. Simply saying “This pipeline is broken, fix it” is a goal, but the agent will likely fix it in ways that are not following your business rules.

​
Process

Process explanation is usually the most important: you will explain how to fix an issue using a defined set of tools, how to get proper context and how to act when facing some unintended cases. What we like to do here is to explain:
  • What is available to the agent: MCPs, CLI tools and scripts
  • How to deal with responses and act based on it
  • Known edge-cases and business rules that are crucial to the task completion
  • Examples for basic tasks and how to solve it the right way
Where difficulty lies in explaining the process is finding the right balance of specializing the agent into solving a task, while keeping the topic broad enough to be able to manage a wide variety of similar tasks. You can explain the process to an agent in order to correctly manage machine via Terraform, but you should avoid being too specific for upgrading some aspects of the machine unless absolutely necessary. You want the agent to see the bigger picture to be able to manage all aspects of a machine via Terraform, not only a couple lines like disk and CPU.

​
Warnings

This last part of the prompt is the place to indicate various counter-intuitive problems the agent can face when troubleshooting, as well as evoking dangerous actions that should not be done. For example, if one of your tools are at a newer version and there’s no way for the agent to know what was updated, you can mention “The method ‘list’ of this tool does not return all fields. You need to query ID individually”. It can also be to prevent dangerous or unnecessary commands, like “Always delete logs that are at least a month old, and keep the recent logs for traceability”.

​
Format

A prompt can be written in various formats, but we encourage structured content rather than plain text. The reason is, more structured the content is, more understandable it becomes, and it applies to both humans and LLMs. We encourage you to write prompts using the industry standard, markdown. Some engineers prefer XML as it can appear more structured for complex tasks.

​
2501 Notes

​
Specialities, Operational Rules and Blacklists

We have deconstructed various aspects of the prompts given to agents in order to obtain better performances and let you have more control/organization over how your agents behave. For a general rule of thumb:
  • Specialities are used to specialize the agent into solving a certain set of tasks. It is meant to be about a specific topic, but broad within that topic to encourage end-to-end resolution and autonomy. It is the main influential factor for an agent’s behavior.
  • Operational Rules are instructions shared across all agents, as well as ways to process for some tasks. It has higher priority than the speciality and can be used to enforce some resolutions in a specific way rather than letting the agent the freedom of solving a problem in various ways.
  • Blacklists are a programmatic way to prevent some commands from being executed. You can still mention prohibited commands to your agents in the specialities, and provide an additional layer of security via Blacklists.

​
Traceability

For traceability and clarity of agent’s task execution over time, we like to tell agents to indicate full path of current location before executing a function via {{ workspace_path }}. Without {{ workspace_path }}: 
# We don't know where file.log is, nor where it is located
file.log | grep "ERROR"
With {{ workspace_path }}: 
# "Precede all commands with {{ workspace_path }}"
cd /myapp/api/logs && file.log | grep "ERROR"

​
OS, tools and Remote Execution Context

It is important to specify the machine’s type to agent, especially when operating on software-specific distribution systems. Omitting some crucial context about the agent’s environment may lead to unreachable tools, installation of unwanted tools, and attempting to reach non-existent locations. In general, you want to indicate OS, Distribution, and possibly version if major changes happened from a version to another that matters to you. Then, indicate the commonly-available tools: some distribution systems (like FortiOS) may not have UNIX or BSD distribution despite being a Linux machine. Finally, make sure you write prompts in the context of how the agent will be used: the tools location, availability and permission may change based on the execution method, workspace or user. For example, an agent executing commands without remote-execution, will most of the time have access to non-UNIX installed CLI tools. However, if remotely executing commands, the user may not be root and don’t have access to the tools unless the full path is mentioned. For an agent with full permissions by default, as the root user: 
sqlplus << EOF
set pagesize 0
set feedback off
select username, con_id from cdb_users where username = 'TABLE_NAME';
exit;
EOF
For an agent remotely executing commands, with a non-root user: 
# .bashrc may not be loaded. Full path necessary, identify as administrator
$ORACLE_HOME/sqlplus -S / as sysdba << EOF
set pagesize 0
set feedback off
select username, con_id from cdb_users where username = 'TABLE_NAME';
exit;
EOF
⌘I