Opponent Modeling
This page is organized as follow:
Table of Contents
Objectives
Power systems are a really important tool today, that can be as resilient as possible to avoid possibly dramatic consequences.
In grid2op, we chose to enforce this property by implementing an “Opponent” (modeled thanks to the BaseOpponent
that can take malicious actions to deteriorate the state of the powergrid and make tha Agent (grid2op.Agent
)
fail. To make the agent “game over” is really easy (for
example it could isolate a load by forcing the disconnection of all the powerline that powers it). This would not be
fair, and that is why the Opponent has some dedicated budget (modeled with the BaseActionBudget
).
The class OpponentSpace
has the delicate role to:
- send the necessary information for the Opponent to attack properly.
- make sure the attack performed by the opponent is legal
- compute the cost of such attack
- make sure this cost is not too high for the opponent budget.
How to create an opponent in any environment
This section is a work in progress, it will only cover how to set up one type of opponent, and supposes that you already know which lines you want to attack, at which frequency etc.
More detailed information about the opponent will be provide in the future.
The set up for the opponent in the “l2rpn_neurips_track1” has the following configuration.
lines_attacked = ["62_58_180", "62_63_160", "48_50_136", "48_53_141", "41_48_131", "39_41_121",
"43_44_125", "44_45_126", "34_35_110", "54_58_154"]
rho_normalization = [0.45, 0.45, 0.6, 0.35, 0.3, 0.2,
0.55, 0.3, 0.45, 0.55]
opponent_attack_cooldown = 12*24 # 24 hours, 1 hour being 12 time steps
opponent_attack_duration = 12*4 # 4 hours
opponent_budget_per_ts = 0.16667 # opponent_attack_duration / opponent_attack_cooldown + epsilon
opponent_init_budget = 144. # no need to attack straightfully, it can attack starting at midday the first day
config = {
"opponent_attack_cooldown": opponent_attack_cooldown,
"opponent_attack_duration": opponent_attack_duration,
"opponent_budget_per_ts": opponent_budget_per_ts,
"opponent_init_budget": opponent_init_budget,
"opponent_action_class": PowerlineSetAction,
"opponent_class": WeightedRandomOpponent,
"opponent_budget_class": BaseActionBudget,
'kwargs_opponent': {"lines_attacked": lines_attacked,
"rho_normalization": rho_normalization,
"attack_period": opponent_attack_cooldown}
}
To create the same type of opponent on the case14 grid you can do:
import grid2op
from grid2op.Action import PowerlineSetAction
from grid2op.Opponent import RandomLineOpponent, BaseActionBudget
env_name = "l2rpn_case14_sandbox"
env_with_opponent = grid2op.make(env_name,
opponent_attack_cooldown=12*24,
opponent_attack_duration=12*4,
opponent_budget_per_ts=0.5,
opponent_init_budget=0.,
opponent_action_class=PowerlineSetAction,
opponent_class=RandomLineOpponent,
opponent_budget_class=BaseActionBudget,
kwargs_opponent={"lines_attacked":
["1_3_3", "1_4_4", "3_6_15", "9_10_12", "11_12_13", "12_13_14"]}
)
# and now you have an opponent on the l2rpn_case14_sandbox
# you can for example
obs = env_with_opponent.reset()
act = ... # chose an action here
obs, reward, done, info = env_with_opponent.step(act)
And for the track2 of neurips, if you want to make it even more complicated, you can add an opponent in the same fashion:
import grid2op
from grid2op.Action import PowerlineSetAction
from grid2op.Opponent import RandomLineOpponent, BaseActionBudget
env_name = "l2rpn_neurips_2020_track2_small"
env_with_opponent = grid2op.make(env_name,
opponent_attack_cooldown=12*24,
opponent_attack_duration=12*4,
opponent_budget_per_ts=0.5,
opponent_init_budget=0.,
opponent_action_class=PowerlineSetAction,
opponent_class=RandomLineOpponent,
opponent_budget_class=BaseActionBudget,
kwargs_opponent={"lines_attacked":
["26_31_106",
"21_22_93",
"17_18_88",
"4_10_162",
"12_14_68",
"14_32_108",
"62_58_180",
"62_63_160",
"48_50_136",
"48_53_141",
"41_48_131",
"39_41_121",
"43_44_125",
"44_45_126",
"34_35_110",
"54_58_154",
"74_117_81",
"80_79_175",
"93_95_43",
"88_91_33",
"91_92_37",
"99_105_62",
"102_104_61"]}
)
# and now you have an opponent on the l2rpn_case14_sandbox
# you can for example
obs = env_with_opponent.reset()
act = ... # chose an action here
obs, reward, done, info = env_with_opponent.step(act)
To summarize what is going on here:
opponent_attack_cooldown: give the minimum number of time between two attacks (here 1 attack per day)
opponent_attack_duration: duration for each attack (when a line is attacked, it will not be possible to reconnect it for that many steps). In the example it’s 4h (so 48 steps)
opponent_action_class: type of the action the opponent will perform (in this case PowerlineSetAction)
opponent_class: type of the opponent. Change it at your own risk.
opponent_budget_class: Each attack will cost some budget to the opponent. If no budget, the opponent cannot attack. This specifies how the budget are computed. Do not change it.
opponent_budget_per_ts: increase of the budget of the opponent per step. The higher this number, the faster the the opponent will regenerate its budget.
opponent_init_budget: initial opponent budget. It is set to 0 to “give” the agent a bit of time before the opponent is triggered.
kwargs_opponent: additional information for the opponent. In this case we provide for each grid the powerline it can attack.
Note
This is only valid for the RandomLineOpponent that disconnect powerlines randomly (but not uniformly!). For other type of Opponent, we don’t provide any information in the documentation at this stage. Feel free to submit a github issue if this is an issue for you.
Detailed Documentation by class
Classes:
|
INTERNAL |
|
|
|
This opponent will disconnect lines randomly among the attackable lines lines_attacked. |
|
This class define an unlimited budget for the opponent. |
|
Is similar to the action space, but for the opponent. |
|
An opponent that disconnect at random any powerlines among a specified list given at the initialization. |
|
This class define an unlimited budget for the opponent. |
|
This opponent will disconnect lines randomly among the attackable lines lines_attacked. |
- class grid2op.Opponent.BaseActionBudget(action_space)[source]
INTERNAL
Warning
/!\ Internal, do not use unless you know what you are doing /!\
This is the base class representing the action budget. It makes sure the opponent uses the correct type of “action”, and compute the bugdet associated to it.
- class grid2op.Opponent.BaseOpponent(action_space)[source]
Methods:
attack
(observation, agent_action, ...)This method is the equivalent of "act" for a regular agent.
get_state
()This function should return the internal state of the Opponent.
init
(partial_env, **kwargs)Generic function used to initialize the derived classes.
reset
(initial_budget)This function is called at the end of an episode, when the episode is over.
set_state
(my_state)This function is used to set the internal state of the Opponent.
tell_attack_continues
(observation, ...)The purpose of this method is to tell the agent that his attack is being continued and to indicate the current state of the grid.
- attack(observation, agent_action, env_action, budget, previous_fails)[source]
This method is the equivalent of “act” for a regular agent.
Opponent, in this framework can have more information than a regular agent (in particular it can view time step t+1), it has access to its current budget etc.
- Parameters
observation (
grid2op.Observation.Observation
) – The last observation (at time t)opp_reward (
float
) – THe opponent “reward” (equivalent to the agent reward, but for the opponent) TODO do i add it back ???done (
bool
) – Whether the game ended or not TODO do i add it back ???agent_action (
grid2op.Action.Action
) – The action that the agent tookenv_action (
grid2op.Action.Action
) – The modification that the environment will take.budget (
float
) – The current remaining budget (if an action is above this budget, it will be replaced by a do nothing.previous_fails (
bool
) – Wheter the previous attack failed (due to budget or ambiguous action)
- Returns
attack (
grid2op.Action.Action
) – The attack performed by the opponent. In this case, a do nothing, all the time.duration (
int
) – The duration of the attack
- get_state()[source]
This function should return the internal state of the Opponent.
This means that after a call to opponent.set_state(opponent.get_state()) the opponent should do the exact same things than without these calls.
- init(partial_env, **kwargs)[source]
Generic function used to initialize the derived classes. For example, if an opponent reads from a file, the path where is the file is located should be pass with this method.
- reset(initial_budget)[source]
This function is called at the end of an episode, when the episode is over. It aims at resetting the self and prepare it for a new episode.
- Parameters
initial_budget (
float
) – The initial budget the opponent has
- set_state(my_state)[source]
This function is used to set the internal state of the Opponent.
- Parameters
my_state –
- tell_attack_continues(observation, agent_action, env_action, budget)[source]
The purpose of this method is to tell the agent that his attack is being continued and to indicate the current state of the grid.
At every time step, either “attack” or “tell_acttack_continues” is called exactly once.
- Parameters
observation (
grid2op.Observation.Observation
) – The last observation (at time t)agent_action (
grid2op.Action.Action
) – The action that the agent tookenv_action (
grid2op.Action.Action
) – The modification that the environment will take.budget (
float
) – The current remaining budget (if an action is above this budget, it will be replaced by a do nothing.
- class grid2op.Opponent.GeometricOpponent(action_space)[source]
This opponent will disconnect lines randomly among the attackable lines lines_attacked. The sampling is done according to the lines load factor (ratio <current going through the line> to <thermal limit of the line>) (see init for more details).
The time of the attack is sampled according to a geometric distribution
Methods:
attack
(observation, agent_action, ...)This method is the equivalent of "attack" for a regular agent.
get_state
()This function should return the internal state of the Opponent.
init
(partial_env[, lines_attacked, ...])Generic function used to initialize the derived classes.
reset
(initial_budget)This function is called at the end of an episode, when the episode is over.
set_state
(my_state)This function is used to set the internal state of the Opponent.
tell_attack_continues
(observation, ...)The purpose of this method is to tell the agent that his attack is being continued and to indicate the current state of the grid.
- attack(observation, agent_action, env_action, budget, previous_fails)[source]
This method is the equivalent of “attack” for a regular agent. Opponent, in this framework can have more information than a regular agent (in particular it can view time step t+1), it has access to its current budget etc. :param observation: The last observation (at time t) :type observation:
grid2op.Observation.Observation
:param opp_reward: THe opponent “reward” (equivalent to the agent reward, but for the opponent) TODO do i add it back ??? :type opp_reward:float
:param done: Whether the game ended or not TODO do i add it back ??? :type done:bool
:param agent_action: The action that the agent took :type agent_action:grid2op.Action.Action
:param env_action: The modification that the environment will take. :type env_action:grid2op.Action.Action
:param budget: The current remaining budget (if an action is above this budget, it will be replaced by a do nothing. :type budget:float
:param previous_fails: Wheter the previous attack failed (due to budget or ambiguous action) :type previous_fails:bool
- Returns
attack (
grid2op.Action.Action
) – The attack performed by the opponent. In this case, a do nothing, all the time.duration (
int
) – The duration of the attack (ifNone
then the attack will be made for the longest allowed time)
- get_state()[source]
This function should return the internal state of the Opponent.
This means that after a call to opponent.set_state(opponent.get_state()) the opponent should do the exact same things than without these calls.
- init(partial_env, lines_attacked=(), attack_every_xxx_hour=24, average_attack_duration_hour=4, minimum_attack_duration_hour=2, pmax_pmin_ratio=4, **kwargs)[source]
Generic function used to initialize the derived classes. For example, if an opponent reads from a file, the path where is the file is located should be pass with this method.
- Parameters
partial_env (grid2op Environment) – A pointer to the environment that initializes the opponent
lines_attacked (
list
) – The list of lines that the XPOpponent should be able to disconnectattack_every_xxx_hour (
float
) –Provide the average duration between two attacks. Note that this should be greater than average_attack_duration_hour as, for now, an agent can only do one consecutive attack. You should provide it in “number of hours” and not in “number of steps”
It is used to compute the attack_hazard_rate. Attacks time are sampled with a duration distribution. For this opponent, we use the simplest of these distributions : The geometric disribution https://en.wikipedia.org/wiki/Geometric_distribution (the discrete time counterpart of the exponential distribution). The attack_hazard_rate is the main parameter of this distribution. It can be seen as the (constant) probability of having an attack in the next step. It is also the inverse of the expectation of the time to an attack.
average_attack_duration_hour (
float
) –Give, in number of hours, the average attack duration. This should be greater than recovery_minimum_duration_hour
Used to compute the recovery_rate: Recovery times are random or at least should have a random part. In our case, we will say that the recovery time is equal to a fixed time (safety procedure time) plus a random time (investigations and repair operations) sampled according to a geometric distribution
minimum_attack_duration_hour (
int
) – Minimum duration of an attack (give it in hour)pmax_pmin_ratio (
float
) – Ratio between the probability of the most likely line to be disconnected and the least likely one.
- reset(initial_budget)[source]
This function is called at the end of an episode, when the episode is over. It aims at resetting the self and prepare it for a new episode.
- Parameters
initial_budget (
float
) – The initial budget the opponent has
- set_state(my_state)[source]
This function is used to set the internal state of the Opponent.
- Parameters
my_state –
- tell_attack_continues(observation, agent_action, env_action, budget)[source]
The purpose of this method is to tell the agent that his attack is being continued and to indicate the current state of the grid.
At every time step, either “attack” or “tell_acttack_continues” is called exactly once.
- Parameters
observation (
grid2op.Observation.Observation
) – The last observation (at time t)agent_action (
grid2op.Action.Action
) – The action that the agent tookenv_action (
grid2op.Action.Action
) – The modification that the environment will take.budget (
float
) – The current remaining budget (if an action is above this budget, it will be replaced by a do nothing.
- class grid2op.Opponent.NeverAttackBudget(action_space)[source]
This class define an unlimited budget for the opponent.
It SHOULD NOT be used if the opponent is allowed to take any actions!
- class grid2op.Opponent.OpponentSpace(compute_budget, init_budget, opponent, attack_duration, attack_cooldown, budget_per_timestep=0.0, action_space=None)[source]
Is similar to the action space, but for the opponent.
This class is used to express some “constraints” on the opponent attack. The opponent is free to attack whatever it wants, for how long it wants and when it wants. This class ensures that the opponent does not break any rules.
- action_space
The action space defining which action the Opponent are allowed to take
- init_budget
The initial budget of the opponent
- Type
float
- compute_budget
The tool used to compute the budget
- Type
grid2op.Opponent.ActionBudget
- opponent
The agent that will take malicious actions.
- previous_fails
Whether the last attack of the opponent failed or not
- Type
bool
- budget_per_timestep
The increase of the opponent budget per time step (if any)
- Type
float
Methods:
attack
(observation, agent_action, env_action)This function calls the attack from the opponent.
close
()if this has a reference to a backend, you need to close it for grid2op to work properly.
has_failed
()This signal is sent by the environment and indicated the opponent attack could not be implmented on the powergrid, most likely due to the attack to be ambiguous.
init_opponent
(partial_env, **kwargs)Generic function used to initialize the opponent.
reset
()Reset the state of the Opponent to its original state, in particular re assign the proper budget to it.
- attack(observation, agent_action, env_action)[source]
This function calls the attack from the opponent.
It check whether the budget is consistent with the attack (budget should be more that the cosst associated with the attack). If the attack cost too much, then it is replaced by a “do nothing” action. Otherwise, the attack will be implemented by the environment.
Note that if the attack is “ambiguous” it will fails (the environment will replace it by a “do nothing” action), but the budget will still be consumed.
NB it is expected that this function update the
OpponentSpace.last_attack
attribute withNone
if the opponent choose not to attack, or with the attack of the opponent otherwise.- Parameters
observation (
grid2op.Observation.Observation
) – The last observation (at time t)agent_action (
grid2op.Action.Action
) – The action that the agent tookenv_action (
grid2op.Action.Action
) – The modification that the environment will take.
- Returns
res – (or “do nothing” if the attack was too costly) or class:NoneType : Returns None if no action is taken
- Return type
grid2op.Action.Action
: The attack the opponent wants to perform
- close()[source]
if this has a reference to a backend, you need to close it for grid2op to work properly. Do not forget to do it.
- has_failed()[source]
This signal is sent by the environment and indicated the opponent attack could not be implmented on the powergrid, most likely due to the attack to be ambiguous.
- class grid2op.Opponent.RandomLineOpponent(action_space)[source]
An opponent that disconnect at random any powerlines among a specified list given at the initialization.
Methods:
attack
(observation, agent_action, ...)This method is the equivalent of "attack" for a regular agent.
init
(partial_env[, lines_attacked])INTERNAL
- attack(observation, agent_action, env_action, budget, previous_fails)[source]
This method is the equivalent of “attack” for a regular agent.
Opponent, in this framework can have more information than a regular agent (in particular it can view time step t+1), it has access to its current budget etc.
- Parameters
observation (
grid2op.Observation.Observation
) – The last observation (at time t)opp_reward (
float
) – THe opponent “reward” (equivalent to the agent reward, but for the opponent) TODO do i add it back ???done (
bool
) – Whether the game ended or not TODO do i add it back ???agent_action (
grid2op.Action.Action
) – The action that the agent tookenv_action (
grid2op.Action.Action
) – The modification that the environment will take.budget (
float
) – The current remaining budget (if an action is above this budget, it will be replaced by a do nothing.previous_fails (
bool
) – Wheter the previous attack failed (due to budget or ambiguous action)
- Returns
attack (
grid2op.Action.Action
) – The attack performed by the opponent. In this case, a do nothing, all the time.duration (
int
) – The duration of the attack (ifNone
then the attack will be made for the longest allowed time)
- class grid2op.Opponent.UnlimitedBudget(action_space)[source]
This class define an unlimited budget for the opponent.
It SHOULD NOT be used if the opponent is allowed to take any actions!
- class grid2op.Opponent.WeightedRandomOpponent(action_space)[source]
This opponent will disconnect lines randomly among the attackable lines lines_attacked. The sampling is weighted by the lines current usage rate divided by some factor rho_normalization (see init for more details).
When an attack becomes possible, the time of the attack will be sampled uniformly in the next attack_period steps (see init).
Methods:
attack
(observation, agent_action, ...)This method is the equivalent of "attack" for a regular agent.
init
(partial_env[, lines_attacked, ...])Generic function used to initialize the derived classes.
reset
(initial_budget)This function is called at the end of an episode, when the episode is over.
tell_attack_continues
(observation, ...)The purpose of this method is to tell the agent that his attack is being continued and to indicate the current state of the grid.
- attack(observation, agent_action, env_action, budget, previous_fails)[source]
This method is the equivalent of “attack” for a regular agent.
Opponent, in this framework can have more information than a regular agent (in particular it can view time step t+1), it has access to its current budget etc.
- Parameters
observation (
grid2op.Observation.Observation
) – The last observation (at time t)opp_reward (
float
) – THe opponent “reward” (equivalent to the agent reward, but for the opponent) TODO do i add it back ???done (
bool
) – Whether the game ended or not TODO do i add it back ???agent_action (
grid2op.Action.Action
) – The action that the agent tookenv_action (
grid2op.Action.Action
) – The modification that the environment will take.budget (
float
) – The current remaining budget (if an action is above this budget, it will be replaced by a do nothing.previous_fails (
bool
) – Wheter the previous attack failed (due to budget or ambiguous action)
- Returns
attack (
grid2op.Action.Action
) – The attack performed by the opponent. In this case, a do nothing, all the time.duration (
int
) – The duration of the attack
- init(partial_env, lines_attacked=[], rho_normalization=[], attack_period=288, **kwargs)[source]
Generic function used to initialize the derived classes. For example, if an opponent reads from a file, the path where is the file is located should be pass with this method.
- Parameters
lines_attacked (
list
) – The list of lines that the WeightedRandomOpponent should be able to disconnectrho_normalization (
list
) – The list of mean usage rates for the attackable lines. Should have the same length as lines_attacked. If no value is given, no normalization will be performed. The weights for sampling the attacked line are rho / rho_normalization.attack_period (
int
) – The number of steps among which the attack may happen. If attack_period=10, then whenever an attack can be made, it will happen in the 10 next steps.
- reset(initial_budget)[source]
This function is called at the end of an episode, when the episode is over. It aims at resetting the self and prepare it for a new episode.
- Parameters
initial_budget (
float
) – The initial budget the opponent has
- tell_attack_continues(observation, agent_action, env_action, budget)[source]
The purpose of this method is to tell the agent that his attack is being continued and to indicate the current state of the grid.
At every time step, either “attack” or “tell_acttack_continues” is called exactly once.
- Parameters
observation (
grid2op.Observation.Observation
) – The last observation (at time t)agent_action (
grid2op.Action.Action
) – The action that the agent tookenv_action (
grid2op.Action.Action
) – The modification that the environment will take.budget (
float
) – The current remaining budget (if an action is above this budget, it will be replaced by a do nothing.
If you still can’t find what you’re looking for, try in one of the following pages:
Still trouble finding the information ? Do not hesitate to send a github issue about the documentation at this link: Documentation issue template