Opponent Modeling 

This page is organized as follow:

Table of Contents

Opponent Modeling

Objectives 

Power systems are a really important tool today, that can be as resilient as possible to avoid possibly dramatic consequences.

In grid2op, we chose to enforce this property by implementing an “Opponent” (modeled thanks to the BaseOpponent that can take malicious actions to deteriorate the state of the powergrid and make tha Agent (grid2op.Agent) fail. To make the agent “game over” is really easy (for example it could isolate a load by forcing the disconnection of all the powerline that powers it). This would not be fair, and that is why the Opponent has some dedicated budget (modeled with the BaseActionBudget).

The class OpponentSpace has the delicate role to: - send the necessary information for the Opponent to attack properly. - make sure the attack performed by the opponent is legal - compute the cost of such attack - make sure this cost is not too high for the opponent budget.

How to create an opponent in any environment 

This section is a work in progress, it will only cover how to set up one type of opponent, and supposes that you already know which lines you want to attack, at which frequency etc.

More detailed information about the opponent will be provide in the future.

The set up for the opponent in the “l2rpn_neurips_track1” has the following configuration.

lines_attacked = ["62_58_180", "62_63_160", "48_50_136", "48_53_141", "41_48_131", "39_41_121",
              "43_44_125", "44_45_126", "34_35_110", "54_58_154"]
rho_normalization = [0.45, 0.45, 0.6, 0.35, 0.3, 0.2,
                     0.55, 0.3, 0.45, 0.55]
opponent_attack_cooldown = 12*24  # 24 hours, 1 hour being 12 time steps
opponent_attack_duration = 12*4  # 4 hours
opponent_budget_per_ts = 0.16667  # opponent_attack_duration / opponent_attack_cooldown + epsilon
opponent_init_budget = 144.  # no need to attack straightfully, it can attack starting at midday the first day
config = {
    "opponent_attack_cooldown": opponent_attack_cooldown,
    "opponent_attack_duration": opponent_attack_duration,
    "opponent_budget_per_ts": opponent_budget_per_ts,
    "opponent_init_budget": opponent_init_budget,
    "opponent_action_class": PowerlineSetAction,
    "opponent_class": WeightedRandomOpponent,
    "opponent_budget_class": BaseActionBudget,
    'kwargs_opponent': {"lines_attacked": lines_attacked,
                        "rho_normalization": rho_normalization,
                        "attack_period": opponent_attack_cooldown}
}

To create the same type of opponent on the case14 grid you can do:

import grid2op
from grid2op.Action import PowerlineSetAction
from grid2op.Opponent import RandomLineOpponent, BaseActionBudget
env_name = "l2rpn_case14_sandbox"

env_with_opponent = grid2op.make(env_name,
                                 opponent_attack_cooldown=12*24,
                                 opponent_attack_duration=12*4,
                                 opponent_budget_per_ts=0.5,
                                 opponent_init_budget=0.,
                                 opponent_action_class=PowerlineSetAction,
                                 opponent_class=RandomLineOpponent,
                                 opponent_budget_class=BaseActionBudget,
                                 kwargs_opponent={"lines_attacked":
                                      ["1_3_3", "1_4_4", "3_6_15", "9_10_12", "11_12_13", "12_13_14"]}
                                 )
# and now you have an opponent on the l2rpn_case14_sandbox
# you can for example
obs = env_with_opponent.reset()

act = ...  # chose an action here
obs, reward, done, info = env_with_opponent.step(act)

And for the track2 of neurips, if you want to make it even more complicated, you can add an opponent in the same fashion:

import grid2op
from grid2op.Action import PowerlineSetAction
from grid2op.Opponent import RandomLineOpponent, BaseActionBudget
env_name = "l2rpn_neurips_2020_track2_small"

env_with_opponent = grid2op.make(env_name,
                                 opponent_attack_cooldown=12*24,
                                 opponent_attack_duration=12*4,
                                 opponent_budget_per_ts=0.5,
                                 opponent_init_budget=0.,
                                 opponent_action_class=PowerlineSetAction,
                                 opponent_class=RandomLineOpponent,
                                 opponent_budget_class=BaseActionBudget,
                                 kwargs_opponent={"lines_attacked":
                                                     ["26_31_106",
                                                      "21_22_93",
                                                      "17_18_88",
                                                      "4_10_162",
                                                      "12_14_68",
                                                      "14_32_108",
                                                      "62_58_180",
                                                      "62_63_160",
                                                      "48_50_136",
                                                      "48_53_141",
                                                      "41_48_131",
                                                      "39_41_121",
                                                      "43_44_125",
                                                      "44_45_126",
                                                      "34_35_110",
                                                      "54_58_154",
                                                      "74_117_81",
                                                      "80_79_175",
                                                      "93_95_43",
                                                      "88_91_33",
                                                      "91_92_37",
                                                      "99_105_62",
                                                      "102_104_61"]}
                                 )
# and now you have an opponent on the l2rpn_case14_sandbox
# you can for example
obs = env_with_opponent.reset()

act = ...  # chose an action here
obs, reward, done, info = env_with_opponent.step(act)

To summarize what is going on here:

opponent_attack_cooldown: give the minimum number of time between two attacks (here 1 attack per day)
opponent_attack_duration: duration for each attack (when a line is attacked, it will not be possible to reconnect it for that many steps). In the example it’s 4h (so 48 steps)
opponent_action_class: type of the action the opponent will perform (in this case PowerlineSetAction)
opponent_class: type of the opponent. Change it at your own risk.
opponent_budget_class: Each attack will cost some budget to the opponent. If no budget, the opponent cannot attack. This specifies how the budget are computed. Do not change it.
opponent_budget_per_ts: increase of the budget of the opponent per step. The higher this number, the faster the the opponent will regenerate its budget.
opponent_init_budget: initial opponent budget. It is set to 0 to “give” the agent a bit of time before the opponent is triggered.
kwargs_opponent: additional information for the opponent. In this case we provide for each grid the powerline it can attack.

Note

This is only valid for the RandomLineOpponent that disconnect powerlines randomly (but not uniformly!). For other type of Opponent, we don’t provide any information in the documentation at this stage. Feel free to submit a github issue if this is an issue for you.

Detailed Documentation by class 

Classes:

`BaseActionBudget`(action_space)	INTERNAL
`BaseOpponent`(action_space)
`GeometricOpponent`(action_space)	This opponent will disconnect lines randomly among the attackable lines lines_attacked.
`NeverAttackBudget`(action_space)	This class define an unlimited budget for the opponent.
`OpponentSpace`(compute_budget, init_budget, ...)	Is similar to the action space, but for the opponent.
`RandomLineOpponent`(action_space)	An opponent that disconnect at random any powerlines among a specified list given at the initialization.
`UnlimitedBudget`(action_space)	This class define an unlimited budget for the opponent.
`WeightedRandomOpponent`(action_space)	This opponent will disconnect lines randomly among the attackable lines lines_attacked.

class grid2op.Opponent.BaseActionBudget(action_space)[source]

INTERNAL

Warning

/!\ Internal, do not use unless you know what you are doing /!\

This is the base class representing the action budget. It makes sure the opponent uses the correct type of “action”, and compute the bugdet associated to it.

class grid2op.Opponent.BaseOpponent(action_space)[source]

Methods:

`attack`(observation, agent_action, ...)	This method is the equivalent of "act" for a regular agent.
`get_state`()	This function should return the internal state of the Opponent.
`init`(partial_env, **kwargs)	Generic function used to initialize the derived classes.
`reset`(initial_budget)	This function is called at the end of an episode, when the episode is over.
`set_state`(my_state)	This function is used to set the internal state of the Opponent.
`tell_attack_continues`(observation, ...)	The purpose of this method is to tell the agent that his attack is being continued and to indicate the current state of the grid.

attack(observation, agent_action, env_action, budget, previous_fails)[source]

This method is the equivalent of “act” for a regular agent.

Opponent, in this framework can have more information than a regular agent (in particular it can view time step t+1), it has access to its current budget etc.

Parameters

observation (grid2op.Observation.Observation) – The last observation (at time t)
opp_reward (float) – THe opponent “reward” (equivalent to the agent reward, but for the opponent) TODO do i add it back ???
done (bool) – Whether the game ended or not TODO do i add it back ???
agent_action (grid2op.Action.Action) – The action that the agent took
env_action (grid2op.Action.Action) – The modification that the environment will take.
budget (float) – The current remaining budget (if an action is above this budget, it will be replaced by a do nothing.
previous_fails (bool) – Wheter the previous attack failed (due to budget or ambiguous action)

Returns

attack (grid2op.Action.Action) – The attack performed by the opponent. In this case, a do nothing, all the time.
duration (int) – The duration of the attack

get_state()[source]

This function should return the internal state of the Opponent.

This means that after a call to opponent.set_state(opponent.get_state()) the opponent should do the exact same things than without these calls.

init(partial_env, **kwargs)[source]: Generic function used to initialize the derived classes. For example, if an opponent reads from a file, the path where is the file is located should be pass with this method.

reset(initial_budget)[source]

This function is called at the end of an episode, when the episode is over. It aims at resetting the self and prepare it for a new episode.

Parameters: initial_budget (float) – The initial budget the opponent has

set_state(my_state)[source]

This function is used to set the internal state of the Opponent.

Parameters: my_state –

tell_attack_continues(observation, agent_action, env_action, budget)[source]

The purpose of this method is to tell the agent that his attack is being continued and to indicate the current state of the grid.

At every time step, either “attack” or “tell_acttack_continues” is called exactly once.

Parameters

observation (grid2op.Observation.Observation) – The last observation (at time t)
agent_action (grid2op.Action.Action) – The action that the agent took
env_action (grid2op.Action.Action) – The modification that the environment will take.
budget (float) – The current remaining budget (if an action is above this budget, it will be replaced by a do nothing.

class grid2op.Opponent.GeometricOpponent(action_space)[source]

This opponent will disconnect lines randomly among the attackable lines lines_attacked. The sampling is done according to the lines load factor (ratio <current going through the line> to <thermal limit of the line>) (see init for more details).

The time of the attack is sampled according to a geometric distribution

Methods:

`attack`(observation, agent_action, ...)	This method is the equivalent of "attack" for a regular agent.
`get_state`()	This function should return the internal state of the Opponent.
`init`(partial_env[, lines_attacked, ...])	Generic function used to initialize the derived classes.
`reset`(initial_budget)	This function is called at the end of an episode, when the episode is over.
`set_state`(my_state)	This function is used to set the internal state of the Opponent.
`tell_attack_continues`(observation, ...)	The purpose of this method is to tell the agent that his attack is being continued and to indicate the current state of the grid.

attack(observation, agent_action, env_action, budget, previous_fails)[source]

This method is the equivalent of “attack” for a regular agent. Opponent, in this framework can have more information than a regular agent (in particular it can view time step t+1), it has access to its current budget etc. :param observation: The last observation (at time t) :type observation: grid2op.Observation.Observation :param opp_reward: THe opponent “reward” (equivalent to the agent reward, but for the opponent) TODO do i add it back ??? :type opp_reward: float :param done: Whether the game ended or not TODO do i add it back ??? :type done: bool :param agent_action: The action that the agent took :type agent_action: grid2op.Action.Action :param env_action: The modification that the environment will take. :type env_action: grid2op.Action.Action :param budget: The current remaining budget (if an action is above this budget, it will be replaced by a do nothing. :type budget: float :param previous_fails: Wheter the previous attack failed (due to budget or ambiguous action) :type previous_fails: bool

Returns

attack (grid2op.Action.Action) – The attack performed by the opponent. In this case, a do nothing, all the time.
duration (int) – The duration of the attack (if None then the attack will be made for the longest allowed time)

get_state()[source]

This function should return the internal state of the Opponent.

This means that after a call to opponent.set_state(opponent.get_state()) the opponent should do the exact same things than without these calls.

init(partial_env, lines_attacked=(), attack_every_xxx_hour=24, average_attack_duration_hour=4, minimum_attack_duration_hour=2, pmax_pmin_ratio=4, **kwargs)[source]

Generic function used to initialize the derived classes. For example, if an opponent reads from a file, the path where is the file is located should be pass with this method.

Parameters

partial_env (grid2op Environment) – A pointer to the environment that initializes the opponent
lines_attacked (list) – The list of lines that the XPOpponent should be able to disconnect
attack_every_xxx_hour (float) –
Provide the average duration between two attacks. Note that this should be greater than average_attack_duration_hour as, for now, an agent can only do one consecutive attack. You should provide it in “number of hours” and not in “number of steps”

It is used to compute the attack_hazard_rate. Attacks time are sampled with a duration distribution. For this opponent, we use the simplest of these distributions : The geometric disribution https://en.wikipedia.org/wiki/Geometric_distribution (the discrete time counterpart of the exponential distribution). The attack_hazard_rate is the main parameter of this distribution. It can be seen as the (constant) probability of having an attack in the next step. It is also the inverse of the expectation of the time to an attack.
average_attack_duration_hour (float) –
Give, in number of hours, the average attack duration. This should be greater than recovery_minimum_duration_hour

Used to compute the recovery_rate: Recovery times are random or at least should have a random part. In our case, we will say that the recovery time is equal to a fixed time (safety procedure time) plus a random time (investigations and repair operations) sampled according to a geometric distribution
minimum_attack_duration_hour (int) – Minimum duration of an attack (give it in hour)
pmax_pmin_ratio (float) – Ratio between the probability of the most likely line to be disconnected and the least likely one.

reset(initial_budget)[source]

This function is called at the end of an episode, when the episode is over. It aims at resetting the self and prepare it for a new episode.

Parameters: initial_budget (float) – The initial budget the opponent has

set_state(my_state)[source]

This function is used to set the internal state of the Opponent.

Parameters: my_state –

tell_attack_continues(observation, agent_action, env_action, budget)[source]

The purpose of this method is to tell the agent that his attack is being continued and to indicate the current state of the grid.

At every time step, either “attack” or “tell_acttack_continues” is called exactly once.

Parameters

observation (grid2op.Observation.Observation) – The last observation (at time t)
agent_action (grid2op.Action.Action) – The action that the agent took
env_action (grid2op.Action.Action) – The modification that the environment will take.
budget (float) – The current remaining budget (if an action is above this budget, it will be replaced by a do nothing.

class grid2op.Opponent.NeverAttackBudget(action_space)[source]

This class define an unlimited budget for the opponent.

It SHOULD NOT be used if the opponent is allowed to take any actions!

class grid2op.Opponent.OpponentSpace(compute_budget, init_budget, opponent, attack_duration, attack_cooldown, budget_per_timestep=0.0, action_space=None)[source]

Is similar to the action space, but for the opponent.

This class is used to express some “constraints” on the opponent attack. The opponent is free to attack whatever it wants, for how long it wants and when it wants. This class ensures that the opponent does not break any rules.

action_space

The action space defining which action the Opponent are allowed to take

Type: grid2op.Action.ActionSpace

init_budget

The initial budget of the opponent

Type: float

compute_budget

The tool used to compute the budget

Type: grid2op.Opponent.ActionBudget

opponent

The agent that will take malicious actions.

Type: grid2op.Opponent.BaseOpponent

previous_fails

Whether the last attack of the opponent failed or not

Type: bool

budget_per_timestep

The increase of the opponent budget per time step (if any)

Type: float

Methods:

`attack`(observation, agent_action, env_action)	This function calls the attack from the opponent.
`close`()	if this has a reference to a backend, you need to close it for grid2op to work properly.
`has_failed`()	This signal is sent by the environment and indicated the opponent attack could not be implmented on the powergrid, most likely due to the attack to be ambiguous.
`init_opponent`(partial_env, **kwargs)	Generic function used to initialize the opponent.
`reset`()	Reset the state of the Opponent to its original state, in particular re assign the proper budget to it.

attack(observation, agent_action, env_action)[source]

This function calls the attack from the opponent.

It check whether the budget is consistent with the attack (budget should be more that the cosst associated with the attack). If the attack cost too much, then it is replaced by a “do nothing” action. Otherwise, the attack will be implemented by the environment.

Note that if the attack is “ambiguous” it will fails (the environment will replace it by a “do nothing” action), but the budget will still be consumed.

NB it is expected that this function update the OpponentSpace.last_attack attribute with None if the opponent choose not to attack, or with the attack of the opponent otherwise.

Parameters

observation (grid2op.Observation.Observation) – The last observation (at time t)
agent_action (grid2op.Action.Action) – The action that the agent took
env_action (grid2op.Action.Action) – The modification that the environment will take.

Returns

res – (or “do nothing” if the attack was too costly) or class:NoneType : Returns None if no action is taken

Return type

grid2op.Action.Action : The attack the opponent wants to perform

close()[source]: if this has a reference to a backend, you need to close it for grid2op to work properly. Do not forget to do it.

has_failed()[source]: This signal is sent by the environment and indicated the opponent attack could not be implmented on the powergrid, most likely due to the attack to be ambiguous.

init_opponent(partial_env, **kwargs)[source]: Generic function used to initialize the opponent. For example, if an opponent reads from a file, the path where is the file is located should be pass with this method.

reset()[source]: Reset the state of the Opponent to its original state, in particular re assign the proper budget to it.

class grid2op.Opponent.RandomLineOpponent(action_space)[source]

An opponent that disconnect at random any powerlines among a specified list given at the initialization.

Methods:

`attack`(observation, agent_action, ...)	This method is the equivalent of "attack" for a regular agent.
`init`(partial_env[, lines_attacked])	INTERNAL

attack(observation, agent_action, env_action, budget, previous_fails)[source]

This method is the equivalent of “attack” for a regular agent.

Opponent, in this framework can have more information than a regular agent (in particular it can view time step t+1), it has access to its current budget etc.

Parameters

observation (grid2op.Observation.Observation) – The last observation (at time t)
opp_reward (float) – THe opponent “reward” (equivalent to the agent reward, but for the opponent) TODO do i add it back ???
done (bool) – Whether the game ended or not TODO do i add it back ???
agent_action (grid2op.Action.Action) – The action that the agent took
env_action (grid2op.Action.Action) – The modification that the environment will take.
budget (float) – The current remaining budget (if an action is above this budget, it will be replaced by a do nothing.
previous_fails (bool) – Wheter the previous attack failed (due to budget or ambiguous action)

Returns

attack (grid2op.Action.Action) – The attack performed by the opponent. In this case, a do nothing, all the time.
duration (int) – The duration of the attack (if None then the attack will be made for the longest allowed time)

init(partial_env, lines_attacked=[], **kwargs)[source]

INTERNAL

Warning

/!\ Internal, do not use unless you know what you are doing /!\ Used when the opponent is created.

Parameters

partial_env –
lines_attacked –
kwargs –

class grid2op.Opponent.UnlimitedBudget(action_space)[source]

This class define an unlimited budget for the opponent.

It SHOULD NOT be used if the opponent is allowed to take any actions!

class grid2op.Opponent.WeightedRandomOpponent(action_space)[source]

This opponent will disconnect lines randomly among the attackable lines lines_attacked. The sampling is weighted by the lines current usage rate divided by some factor rho_normalization (see init for more details).

When an attack becomes possible, the time of the attack will be sampled uniformly in the next attack_period steps (see init).

Methods:

`attack`(observation, agent_action, ...)	This method is the equivalent of "attack" for a regular agent.
`init`(partial_env[, lines_attacked, ...])	Generic function used to initialize the derived classes.
`reset`(initial_budget)	This function is called at the end of an episode, when the episode is over.
`tell_attack_continues`(observation, ...)	The purpose of this method is to tell the agent that his attack is being continued and to indicate the current state of the grid.

attack(observation, agent_action, env_action, budget, previous_fails)[source]

This method is the equivalent of “attack” for a regular agent.

Opponent, in this framework can have more information than a regular agent (in particular it can view time step t+1), it has access to its current budget etc.

Parameters

observation (grid2op.Observation.Observation) – The last observation (at time t)
opp_reward (float) – THe opponent “reward” (equivalent to the agent reward, but for the opponent) TODO do i add it back ???
done (bool) – Whether the game ended or not TODO do i add it back ???
agent_action (grid2op.Action.Action) – The action that the agent took
env_action (grid2op.Action.Action) – The modification that the environment will take.
budget (float) – The current remaining budget (if an action is above this budget, it will be replaced by a do nothing.
previous_fails (bool) – Wheter the previous attack failed (due to budget or ambiguous action)

Returns

attack (grid2op.Action.Action) – The attack performed by the opponent. In this case, a do nothing, all the time.
duration (int) – The duration of the attack

init(partial_env, lines_attacked=[], rho_normalization=[], attack_period=288, **kwargs)[source]

Generic function used to initialize the derived classes. For example, if an opponent reads from a file, the path where is the file is located should be pass with this method.

Parameters

lines_attacked (list) – The list of lines that the WeightedRandomOpponent should be able to disconnect
rho_normalization (list) – The list of mean usage rates for the attackable lines. Should have the same length as lines_attacked. If no value is given, no normalization will be performed. The weights for sampling the attacked line are rho / rho_normalization.
attack_period (int) – The number of steps among which the attack may happen. If attack_period=10, then whenever an attack can be made, it will happen in the 10 next steps.

reset(initial_budget)[source]

This function is called at the end of an episode, when the episode is over. It aims at resetting the self and prepare it for a new episode.

Parameters: initial_budget (float) – The initial budget the opponent has

tell_attack_continues(observation, agent_action, env_action, budget)[source]

The purpose of this method is to tell the agent that his attack is being continued and to indicate the current state of the grid.

At every time step, either “attack” or “tell_acttack_continues” is called exactly once.

Parameters

observation (grid2op.Observation.Observation) – The last observation (at time t)
agent_action (grid2op.Action.Action) – The action that the agent took
env_action (grid2op.Action.Action) – The modification that the environment will take.
budget (float) – The current remaining budget (if an action is above this budget, it will be replaced by a do nothing.

If you still can’t find what you’re looking for, try in one of the following pages:

Still trouble finding the information ? Do not hesitate to send a github issue about the documentation at this link: Documentation issue template

Opponent Modeling

Objectives

How to create an opponent in any environment

Detailed Documentation by class

Opponent Modeling 

Objectives 

How to create an opponent in any environment 

Detailed Documentation by class 