Agones – Guides

Docs: Feature Stages

Thu, 26 Sep 2019 01:20:41 +0000

Supported Versions

Agones versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version , following Semantic Versioning terminology.

Agones Features

A feature within Agones can be in Alpha, Beta or Stable stage.

Feature Gates

Alpha and Beta features can be enabled or disabled through the agones.featureGates configuration option that can be found in the Helm configuration documentation.

The current set of alpha and beta feature gates are:

Feature Name	Gate	Default	Stage	Since
Multicluster Allocation^*	N/A	Enabled	`Beta`	1.6.0
Example Gate (not in use)	`Example`	Disabled	None	0.13.0
Port Allocations to Multiple Containers	`ContainerPortAllocation`	Enabled	`Beta`	1.7.0
Player Tracking	`PlayerTracking`	Disabled	`Alpha`	1.6.0

^*Multicluster Allocation was started before this process was in place, and therefore does not have a feature gate and cannot be disabled.

Description of Stages

Alpha

An Alpha feature means:

Disabled by default.
Might be buggy. Enabling the feature may expose bugs.
Support for this feature may be dropped at any time without notice.
The API may change in incompatible ways in a later software release without notice.
Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.

Note

Please do try Alpha features and give feedback on them. This is important to ensure less breaking changes through the Beta period.

Beta

A Beta feature means:

Enabled by default, but able to be disabled through a feature gate.
The feature is well tested. Enabling the feature is considered safe.
Support for the overall feature will not be dropped, though details may change.
The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable releases. When this happens, we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature.
Recommended for only non-business-critical uses because of potential for incompatible changes in subsequent releases. If you have multiple clusters that can be upgraded independently, you may be able to relax this restriction.

Note

Note: Please do try Beta features and give feedback on them! After they exit beta, it may not be practical for us to make more changes.

Stable

A Stable feature means:

The feature is enabled and the corresponding feature gate no longer exists.
Stable versions of features will appear in released software for many subsequent versions.

Feature Stage Indicators

There are a variety of features with Agones, how can we determine what stage each feature is in?

Below are indicators for each type of functionality that can be used to determine the feature stage for a given aspect of Agones.

Custom Resource Definitions (CRDs)

This refers to Kubernetes resource for Agones, such as GameServer, Fleet and GameServerAllocation.

New CRDs

For new resources, the stage of the resource will be indicated by the apiVersion of the resource.

For example: apiVersion: "agones.dev/v1" is a stable resource, apiVersion: "agones.dev/v1beta1" is a beta stage resource, and apiVersion: "agones.dev/v1alpha1" is an alpha stage resource.

New CRD attributes

When alpha and beta attributes are added to an existing stable Agones CRD, we will follow the Kubernetes Adding Unstable Features to Stable Versions Guide to optimise on the least amount of breaking changes for users as attributes progress through feature stages.

alpha and beta attributes will be added to the existing CRD as optional and documented with their feature stage. Attempting to populate these alpha and beta attributes on an Agones CRD will return a validation error if their accompanying Feature Flag is not enabled.

alpha and beta attributes can be subject to change of name and structure, and will result in breaking changes before moving to a stable stage. These changes will be outlined in release notes and feature documentation.

Agones Game Server SDK

Any alpha or beta Game Server SDK functionality will be a subpackage of the sdk package. For example , functionality found in a sdk.alphav1 package should be considered at the alpha feature stage.

Only experimental functionality will be found in any alpha and beta SDK packages, and as such may change as development occurs.

As SDK features move to through feature stages towards stable, the previous version of the SDK API will remain for at least one release to enable easy migration to the more stable feature stage (i.e. from alpha -> beta, beta -> stable)

Any other SDK functionality not marked as alpha or beta is assumed to be stable.

REST & gRPC APIs

REST and gRPC API will have versioned paths where appropriate to indicate their feature stage.

For example, a REST API with a prefix of v1alpha1 is an alpha stage feature: http://api.example.com/v1alpha1/exampleaction.

Similar to the SDK, any alpha or beta gRPC functionality will be a subpackage of the main API package. For example, functionality found in a api.alphav1 package should be considered at the alpha feature stage.

Docs: Agones Game Server Client SDKs

Wed, 02 Jan 2019 10:16:30 +0000

Overview

The client SDKs are required for a game server to work with Agones.

The current supported SDKs are:

The SDKs are relatively thin wrappers around gRPC generated clients, or an implementation of the REST API (exposed via grpc-gateway), where gRPC client generation and compilation isn’t well supported.

They connect to a small process that Agones coordinates to run alongside the Game Server in a Kubernetes Pod. This means that more languages can be supported in the future with minimal effort (but pull requests are welcome! 😊 ).

There is also local development tooling for working against the SDK locally, without having to spin up an entire Kubernetes infrastructure.

Connecting to the SDK Server

Starting with Agones 1.1.0, the port that the SDK Server listens on for incoming gRPC or HTTP requests is configurable. This provides flexibility in cases where the default port conflicts with a port that is needed by the game server.

Agones will automatically set the following environment variables on all game server containers:

AGONES_SDK_GRPC_PORT: The port where the gRPC server is listening (defaults to 9357)
AGONES_SDK_HTTP_PORT: The port where the grpc-gateway is listening (defaults to 9358)

The SDKs will automatically discover and connect to the gRPC port specified in the environment variable.

If your game server requires using a REST client, it is advised to use the port from the environment variable, otherwise your game server will not be able to contact the SDK server if it is configured to use a non-default port.

Function Reference

While each of the SDKs are canonical to their languages, they all have the following functions that implement the core responsibilities of the SDK.

For language specific documentation, have a look at the respective source (linked above), and the examples .

Calling any of state changing functions mentioned below does not guarantee that GameServer Custom Resource object would actually change its state right after the call. For instance, it could be moved to the Shutdown state elsewhere (for example, when a fleet scales down), which leads to no changes in GameServer object. You can verify the result of this call by waiting for the desired state in a callback to WatchGameServer() function.

Functions which changes GameServer state or settings are:

Ready()
Shutdown()
SetLabel()
SetAnnotation()
Allocate()
Reserve()
Alpha().SetCapacity()
Alpha().PlayerConnect()
Alpha().PlayerDisconnect()

Lifecycle Management

Ready()

This tells Agones that the Game Server is ready to take player connections. Once a Game Server has specified that it is Ready, then the Kubernetes GameServer record will be moved to the Ready state, and the details for its public address and connection port will be populated.

While Agones prefers that Shutdown() is run once a game has completed to delete the GameServer instance, if you want or need to move an Allocated GameServer back to Ready to be reused, you can call this SDK method again to do this.

Health()

This sends a single ping to designate that the Game Server is alive and healthy. Failure to send pings within the configured thresholds will result in the GameServer being marked as Unhealthy.

See the gameserver.yaml for all health checking configurations.

Reserve(seconds)

With some matchmaking scenarios and systems it is important to be able to ensure that a GameServer is unable to be deleted, but doesn’t trigger a FleetAutoscaler scale up. This is where Reserve(seconds) is useful.

Reserve(seconds) will move the GameServer into the Reserved state for the specified number of seconds (0 is forever), and then it will be moved back to Ready state. While in Reserved state, the GameServer will not be deleted on scale down or Fleet update, and also it could not be Allocated using GameServerAllocation.

This is often used when a game server process must register itself with an external system, such as a matchmaker, that requires it to designate itself as available for a game session for a certain period. Once a game session has started, it should call SDK.Allocate() to designate that players are currently active on it.

Calling other state changing SDK commands such as Ready or Allocate will turn off the timer to reset the GameServer back to the Ready state or to promote it to an Allocated state accordingly.

Allocate()

With some matchmakers and game matching strategies, it can be important for game servers to mark themselves as Allocated. For those scenarios, this SDK functionality exists.

There is a chance that GameServer does not actually become Allocated after this call. Please refer to the general note in Function Reference above.

Note

Using a GameServerAllocation is preferred in all other scenarios, as it gives Agones control over how packed GameServers are scheduled within a cluster, whereas with Allocate() you relinquish control to an external service which likely doesn’t have as much information as Agones.

Shutdown()

This tells Agones to shut down the currently running game server. The GameServer state will be set Shutdown and the backing Pod will be deleted, if they have not shut themselves down already.

Configuration Retrieval

GameServer()

This returns most of the backing GameServer configuration and Status. This can be useful for instances where you may want to know Health check configuration, or the IP and Port the GameServer is currently allocated to.

Since the GameServer contains an entire PodTemplate the returned object is limited to that configuration that was deemed useful. If there are areas that you feel are missing, please file an issue or pull request.

The easiest way to see what is exposed, is to check the sdk.proto , specifically at the message GameServer.

For language specific documentation, have a look at the respective source (linked above), and the examples .

WatchGameServer(function(gameserver){…})

This executes the passed in callback with the current GameServer details whenever the underlying GameServer configuration is updated. This can be useful to track GameServer > Status > State changes, metadata changes, such as labels and annotations, and more.

In combination with this SDK, manipulating Annotations and Labels can also be a useful way to communicate information through to running game server processes from outside those processes. This is especially useful when combined with GameServerAllocation applied metadata.

The easiest way to see what is exposed, is to check the sdk.proto , specifically at the message GameServer.

For language specific documentation, have a look at the respective source (linked above), and the examples .

Metadata Management

SetLabel(key, value)

This will set a Label value on the backing GameServer record that is stored in Kubernetes. To maintain isolation, the key value is automatically prefixed with “agones.dev/sdk-”

Warning

There are limits on the characters that be used for label keys and values. Details are here.

This can be useful if you want information from your running game server process to be observable or searchable through the Kubernetes API.

SetAnnotation(key, value)

This will set a Annotation value on the backing Gameserver record that is stored in Kubernetes. To maintain isolation, the key value is automatically prefixed with “agones.dev/sdk-”

This can be useful if you want information from your running game server process to be observable through the Kubernetes API.

Player Tracking

Warning

The Player Tracking feature is currently Alpha, not enabled by default, and may change in the future.

Use the Feature Gate PlayerTracking to enable and test this feature.

See the Feature Gate documentation for details on how to enable features.

Alpha().PlayerConnect(playerID)

This function increases the SDK’s stored player count by one, and appends this playerID to GameServer.Status.Players.IDs.

GameServer.Status.Players.Count and GameServer.Status.Players.IDs are then set to update the player count and id list a second from now, unless there is already an update pending, in which case the update joins that batch operation.

PlayerConnect() returns true and adds the playerID to the list of playerIDs if this playerID was not already in the list of connected playerIDs.

If the playerID exists within the list of connected playerIDs, PlayerConnect() will return false, and the list of connected playerIDs will be left unchanged.

An error will be returned if the playerID was not already in the list of connected playerIDs but the player capacity for the server has been reached. The playerID will not be added to the list of playerIDs.

Note

Do not use this method if you are manually managing GameServer.Status.Players.IDs and GameServer.Status.Players.Count through the Kubernetes API, as indeterminate results will occur.

Alpha().PlayerDisconnect(playerID)

This function decreases the SDK’s stored player count by one, and removes the playerID from GameServer.Status.Players.IDs.

PlayerDisconnect() will return true and remove the supplied playerID from the list of connected playerIDs if the playerID value exists within the list.

If the playerID was not in the list of connected playerIDs, the call will return false, and the connected playerID list will be left unchanged.

Note

Do not use this method if you are manually managing GameServer.Status.Players.IDs and GameServer.Status.Players.Count through the Kubernetes API, as indeterminate results will occur.

Alpha().SetPlayerCapacity(count)

Update the GameServer.Status.Players.Capacity value with a new capacity.

Alpha().GetPlayerCapacity()

This function retrieves the current player capacity. This is always accurate from what has been set through this SDK, even if the value has yet to be updated on the GameServer status resource.

Note

If GameServer.Status.Players.Capacity is set manually through the Kubernetes API, use SDK.GameServer() or SDK.WatchGameServer() instead to view this value.

Alpha().GetPlayerCount()

This function returns if the playerID is currently connected to the GameServer. This is always accurate from what has been set through this SDK, even if the value has yet to be updated on the GameServer status resource.

Note

If GameServer.Status.Players.IDs is set manually through the Kubernetes API, use SDK.GameServer() or SDK.WatchGameServer() instead to retrieve the current player count.

Alpha().IsPlayerConnected(playerID)

Note

If GameServer.Status.Players.IDs is set manually through the Kubernetes API, use SDK.GameServer() or SDK.WatchGameServer() instead to determine connected status.

Alpha().GetConnectedPlayers()

This function returns the list of the currently connected player ids. This is always accurate from what has been set through this SDK, even if the value has yet to be updated on the GameServer status resource.

Note

If GameServer.Status.Players.IDs is set manually through the Kubernetes API, use SDK.GameServer() or SDK.WatchGameServer() instead to list the connected players.

Writing your own SDK

If there isn’t an SDK for the language and platform you are looking for, you have several options:

gRPC Client Generation

If client generation is well supported by gRPC, then generate client(s) from the proto files found in the proto/sdk , directory and look at the current sdks to see how the wrappers are implemented to make interaction with the SDK server simpler for the user.

REST API Implementation

If client generation is not well supported by gRPC, or if there are other complicating factors, implement the SDK through the REST HTTP+JSON interface. This could be written by hand, or potentially generated from the Swagger/OpenAPI Specifications .

Finally, if you build something that would be usable by the community, please submit a pull request!

SDK Conformance Test

There is a tool SDK server Conformance checker which will run Local SDK server and record all requests your client is performing.

In order to check that SDK is working properly you should write simple SDK test client which would use all methods of your SDK.

Also to test that SDK client is receiving valid Gameserver data, your binary should set the same Label value as creation timestamp which you will receive as a result of GameServer() call and Annotation value same as gameserver UID received by Watch gameserver callback.

Complete list of endpoints which should be called by your test client is the following:

ready,allocate,setlabel,setannotation,gameserver,health,shutdown,watch

In order to run this test SDK server locally use:

SECONDS=30 make run-sdk-conformance-local

Docker container would timeout in 30 seconds and give your the comparison of received requests and expected requests

For instance you could run Go SDK conformance test and see how the process goes:

SDK_FOLDER=go make run-sdk-conformance-test

In order to add test client for your SDK, write sdktest.sh and Dockerfile. Refer to Golang SDK Conformance testing directory structure .

Building the Tools

If you wish to build the binaries from source the make target build-agones-sdk-binary will compile the necessary binaries for all supported operating systems (64 bit windows, linux and osx).

You can find the binaries in the bin folder in `cmd/sdk-server` once compilation is complete.

See Developing, Testing and Building Agones for more details.

Docs: GameServer Creation, Allocation and Shutdown Lifecycle

Fri, 01 Feb 2019 02:15:18 +0000

Matchmaker requests a GameServer from a Fleet

This is the preferred workflow for a GameServer, in which an external matchmaker requests an allocation from one or more Fleets using a GameServerAllocation:

Matchmaker requires game server process registration

Scenarios in which a Matchmaker requires a game server process to register themselves with the matchmaker, and the matchmaker decides which GameServer players are sent to, this flow is common:

Warning

This does relinquish control over how GameServers are packed across the cluster to the external matchmaker. It is likely it will not do as good a job at packing and scaling as Agones.

Next Steps:

Read the various references, including the GameServer and Fleet reference materials.
Review the specifics of Health Checking.
See all the commands the Client SDK provides - we only show a few here!
If you aren’t familiar with the term Pod, this should provide a reference.

Docs: Fleet Updates

Tue, 27 Aug 2019 03:58:19 +0000

Rolling Update Strategy

When Fleets are edited and updated, the default strategy of Agones is to roll the new version of the GameServer out to the entire Fleet, in a step by step increment and decrement by adding a chunk of the new version and removing a chunk of the current set of GameServers.

This is done while ensuring that Allocated GameServers are not deleted until they are specifically shutdown through the game servers SDK, as they are expected to have players on them.

You can see this in the Fleet.Spec.Strategy reference, with controls for how much of the Fleet is incremented and decremented at one time:

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%

So when a Fleet is edited (any field other than replicas, see note below), either through kubectl edit/apply or via the Kubernetes API, this performs the following operations:

Adds the maxSurge number of GameServers to the Fleet.
Shutdown the maxUnavailable number of GameServers in the Fleet, skipping Allocated GameServers.
Repeat above steps until all the previous GameServer configurations have been Shutdown and deleted.

Note

When Fleet update contains only changes to the replicas parameter, then new GameServers will be created/deleted straight away, which means in that case maxSurge and maxUnavailable parameters for a RollingUpdate will not be used. The RollingUpdate strategy takes place when you update spec parameters other than replicas.

If you are using a Fleet which is scaled by a FleetAutoscaler, read the Fleetautoscaler guide for more details on how RollingUpdates with FleetAutoscalers need to be implemented.

You could also check the behaviour of the Fleet with a RollingUpdate strategy on a test Fleet to preview your upcoming updates. Use kubectl describe fleet to track scaling events in a Fleet.

Recreate Strategy

This is an optimal Fleet update strategy if you want to replace all GameServers that are not Allocated with a new version as quickly as possible.

You can see this in the Fleet.Spec.Strategy reference:

  strategy:
    type: Recreate

So when a Fleet is edited, either through kubectl edit/apply or via the Kubernetes API, this performs the following operations:

Shutdown all GameServers in the Fleet that are not currently Allocated.
Create the same number of the new version of the GameServers that were previously deleted.
Repeat above steps until all the previous GameServer configurations have been Shutdown and deleted.

Two (or more) Fleets Strategy

If you want very fine-grained control over the rate that new versions of a GameServer configuration is rolled out, or if you want to do some version of A/B testing or smoke test between different versions, running two (or more) Fleets at the same time is a good solution for this.

To do this, create a second Fleet inside your cluster, starting with zero replicas. From there you can scale this newer Fleet up and the older Fleet down as required by your specific rollout strategy.

This also allows you to rollback if issues arise with the newer version, as you can delete the newer Fleet and scale up the old Fleet to its previous levels, resulting in minimal impact to the players.

Note

For GameServerAllocation, you will need to have at least a single shared label between the GameServers in each Fleet.

GameServerAllocation Across Fleets

Since GameServerAllocation is powered by label selectors, it is possible to allocate across multiple fleets, and/or give preference to particular sets of GameServers over others. You can see details of this in the GameServerAllocation reference.

In a scenario where a new v2 version of a Fleet is being slowly scaled up in a separate Fleet from the previous v1 Fleet, we can specify that we prefer allocation to occur from the v2 Fleet, and if none are available, fallback to the v1 Fleet, like so:

apiVersion: "allocation.agones.dev/v1"
kind: GameServerAllocation
spec:
  required:
    matchLabels:
      game: my-awesome-game
  preferred:
    - matchLabels:
        agones.dev/fleet: v2

In this example, all GameServers have the label game: my-awesome-game, so the Allocation will search across both Fleets through that mechanism. The preferred label matching selector tells the allocation system to first search all GameServers with the v2 Fleet label, and if not found, search through the rest of the set.

The above GameServerAllocation can then be used while you scale up the v2 Fleet and scale down the original Fleet at the rate that you deem fit for your specific rollout.

Docs: GameServer Health Checking

Thu, 03 Jan 2019 01:20:19 +0000

Disabling Health Checking

By default, health checking is enabled, but it can be turned off by setting the spec.health.disabled property to true.

SDK API

The Health() function on the SDK object needs to be called at an interval less than the spec.health.periodSeconds threshold time to be considered before it will be considered a failure.

The health check will also need to have not been called a consecutive number of times (spec.health.failureTheshold), giving it a chance to heal if it there is an issue.

Health Failure Strategy

The following is the process for what happens to a GameServer when it is unhealthy.

If the GameServer container exits with an error before the GameServer moves to Ready then, it is restarted as per the restartPolicy (which defaults to “Always”).
If the GameServer fails health checking at any point, then it doesn’t restart, but moves to an Unhealthy state.
If the GameServer container exits while in Ready, Allocated or Reserved state, it will be restarted as per the restartPolicy (which defaults to “Always”, since RestartPolicy is a Pod wide setting), but will immediately move to an Unhealthy state.
If the SDK sidecar fails, then it will be restarted, assuming the RestartPolicy is Always/OnFailure.

Fleet Management of Unhealthy GameServers

If a GameServer moves into an Unhealthy state when it is not part of a Fleet, the GameServer will remain in the Unhealthy state until explicitly deleted. This is useful for debugging Unhealthy GameServers, or if you are creating your own GameServer management layer, you can explicitly choose what to do if a GameServer becomes Unhealthy.

If a GameServer is part of a Fleet, the Fleet management system will delete any Unhealthy GameServers and immediately replace them with a brand new GameServer to ensure it has the configured number of Replicas.

Configuration Reference

  # Health checking for the running game server
  health:
    # Disable health checking. defaults to false, but can be set to true
    disabled: false
    # Number of seconds after the container has started before health check is initiated. Defaults to 5 seconds
    initialDelaySeconds: 5
    # If the `Health()` function doesn't get called at least once every period (seconds), then
    # the game server is not healthy. Defaults to 5
    periodSeconds: 5
    # Minimum consecutive failures for the health probe to be considered failed after having succeeded.
    # Defaults to 3. Minimum value is 1
    failureThreshold: 3

See the full GameServer example for more details

Example

C++

For a configuration that requires a health ping every 5 seconds, the example below sends a request every 2 seconds to be sure that the GameServer is under the threshold.

void doHealth(agones::SDK *sdk) {
    while (true) {
        if (!sdk->Health()) {
            std::cout << "Health ping failed" << std::endl;
        } else {
            std::cout << "Health ping sent" << std::endl;
        }
        std::this_thread::sleep_for(std::chrono::seconds(2));
    }
}

int main() {
    agones::SDK *sdk = new agones::SDK();
    bool connected = sdk->Connect();
    if (!connected) {
        return -1;
    }
    std::thread health (doHealth, sdk);

    // ...  run the game server code

}

Full Game Server

Also look in the examples directory.

Docs: Player Tracking

Tue, 19 May 2020 00:00:00 +0000

Warning

The Player Tracking feature is currently Alpha, not enabled by default, and may change in the future.

Use the Feature Gate PlayerTracking to enable and test this feature.

See the Feature Gate documentation for details on how to enable features.

Managing GameServer Capacities

To track your GameServer current player capacity, Agones gives you the ability to both set an initial capacity at GameServer creation, as well be able to change it during the lifecycle of the GameServer through the Agones SDK.

To set the initial capacity, you can do so via GameServer.Spec.Players.InitialCapacity like so:

apiVersion: "agones.dev/v1"
kind: GameServer
metadata:
  name: "gs-example"
spec:
  # ...
  players:
    # set this GameServer's initial player capacity to 10
    initialCapacity: 10

From there, if you need to change the capacity of the GameSever as gameplay is in progress, you can also do so via SDK.Alpha().SetPlayerCapacity(count)

The current player capacity is represented in GameServer.Status.Players.Capacity resource value.

We can see this in action, when we look at the Status section of a GameServer resource , wherein the capacity has been set to 20:

...
Status:
  Address:    14.81.195.72
  Node Name:  gke-test-cluster-default-6cd0ba67-1mps
  Players:
    Capacity:  20
    Count:     0
    Ids:       <nil>
  Ports:
    Name:          gameport
    Port:          7983
  Reserved Until:  <nil>
  State:           Ready

From the SDK, the game server binary can also retrieve the current player capacity via SDK.Alpha().GetPlayerCapacity().

Note

Changing the capacity value here has no impact on players actually connected to or trying to connect to your server, as that is not a responsibility of Agones.

This functionality is for tracking purposes only.

Connecting and Disconnecting Players

As players connect and disconnect from your game, the Player Tracking functions enable you to track which players are currently connected.

It assumed that each player that connects has a unique token that identifies them as a player.

When a player connects to the game server binary, calling SDK.Alpha().PlayerConnect(playerID) with the unique player token will register them as connected, and store their player id.

At disconnection time, call SDK.Alpha().PlayerDisconnect(playerID) , which will deregister them and remove their player id from the list.

Each of these playerIDs is stored on GameServer.Status.Players.IDs, and the current count of connected players can be seen in GameServer.Status.Players.Count.

You can see this in action below in the GameServer Status section, where there are 4 players connected:

...
Status:
  Address:    39.82.196.74
  Node Name:  gke-test-cluster-default-6cd0ba77-1mps
  Players:
    Capacity:  10
    Count:     4
    Ids:
      xy8a
      m0ux
      71nj
      lpq5
  Ports:
    Name:          gameport
    Port:          7166
  Reserved Until:  <nil>
  State:           Ready

Note

Calling PlayerConnect or PlayerDisconnect functions will not connect or disconnect players, as that is not under the control of Agones.

This functionality is for tracking purposes only.

Checking Player Data

Not only is the connected player data stored on the GameServer resource, it is also stored in memory within the SDK, so that it can be used from within the game server binary as a realtime, thread safe, registry of connected players.

Therefore, if you want to:

Get the current player count, call SDK.Alpha().GetPlayerCount()
Check if a specific player is connected, call SDK.Alpha().IsPlayerConnected(playerID)
Retrieve the full list of connected players, call SDK.Alpha().GetConnectedPlayers()

Next Steps

Review the Player Tracking SDK Reference

Docs: Local Game Server

Tue, 19 Feb 2019 08:00:00 +0000

You can register a local game server with Agones. This means you can run an experimental build of your game server in the Agones environment without the need of packaging and deploying it to a fleet. This allows you to quickly iterate on your game server code while still being able to plugin to your Agones environment.

Register your server with Agones

To register your local game server you’ll need to know the IP address of the machine running it and the port. With that you’ll create a game server config like the one below.

apiVersion: "agones.dev/v1"
kind: GameServer
metadata:
  name: my-local-server
  annotations:
    # Causes Agones to register your local game server at 192.1.1.2, replace with your server's IP address.
    agones.dev/dev-address: "192.1.1.2"
spec:
  ports:
  - name: default
    portPolicy: Static
    hostPort: 17654
    containerPort: 17654
  # The following is ignored but required due to validation.
  template:
    spec:
      containers:
      - name: simple-udp
        image: gcr.io/agones-images/udp-server:0.21

Once you save this to a file make sure you have kubectl configured to point to your Agones cluster and then run kubectl apply -f dev-gameserver.yaml. This will register your server with Agones.

Local Game Servers has a few limitations:

PortPolicy must be Static.
The game server is not managed by Agones. Features like autoscaling, replication, etc are not available.

When you are finished working with your server, you can remove the registration with kubectl delete -f dev-gameserver.yaml

Docs: Latency Testing with Multiple Clusters

Thu, 03 Jan 2019 01:20:30 +0000

To make latency testing easier, Agones installs with a simple ping service with both HTTP and UDP services that can be called for the purpose of timing how long the roundtrip takes for information to be returned from either of these services.

Installing

By default, Agones installs Kubernetes Services for both HTTP and the UDP ping endpoints. These can be disabled entirely, or disabled individually. See the Helm install guide for the parameters to pass through, as well as configuration options.

The ping services as all installed under the agones-system namespace.

HTTP Service

This exposes an endpoint that returns a simple text HTTP response on request to the root “/” path. By default this is ok, but it can be configured via the agones.ping.http.response parameter.

This could be useful for providing clusters with unique lookup names, such that clients are able to identify clusters from their responses.

To lookup the details of this service, run kubectl describe service agones-ping-http-service --namespace=agones-system

UDP Service

The UDP ping service is a rate limited UDP echo service that returns the udp packet that it receives to its designated sender.

Since UDP sender details can be spoofed, this service is rate limited to 20 requests per second, per sender address, per running instance (default is 2).

This rate limit can be raised or lowered via the Helm install parameter agones.ping.udp.rateLimit.

UDP packets are also limited to 1024 bytes in size.

To lookup the details of this service, run kubectl describe service agones-ping-udp-service --namespace=agones-system

Client side tooling

We deliberately didn’t provide any game client libraries, as all major languages and engines have capabilities to send HTTP requests as well as UDP packets.

Docs: Metrics

Thu, 03 Jan 2019 03:58:19 +0000

Agones controller exposes metrics via OpenCensus. OpenCensus is a single distribution of libraries that collect metrics and distributed traces from your services, we only use it for metrics but it will allow us to support multiple exporters in the future.

We choose to start with Prometheus as this is the most popular with Kubernetes but it is also compatible with Stackdriver. If you need another exporter, check the list of supported exporters. It should be pretty straightforward to register a new one. (GitHub PRs are more than welcome.)

We plan to support multiple exporters in the future via environment variables and helm flags.

Backend integrations

Prometheus

If you are running a Prometheus instance you just need to ensure that metrics and kubernetes service discovery are enabled. (helm chart values agones.metrics.prometheusEnabled and agones.metrics.prometheusServiceDiscovery). This will automatically add annotations required by Prometheus to discover Agones metrics and start collecting them. (see example)

Prometheus Operator

If you have Prometheus operator installed in your cluster, make sure to add a ServiceMonitor to discover Agones metrics as shown below:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: agones
  labels:
    app: agones
spec:
  selector:
    matchLabels:
        agones.dev/role: controller
  endpoints:
  - port: web

Finally include that ServiceMonitor in your Prometheus instance CRD, this is usually done by adding a label to the ServiceMonitor above that is matched by the prometheus instance of your choice.

Stackdriver

We support the OpenCensus Stackdriver exporter. In order to use it you should enable Stackdriver Monitoring API in Google Cloud Console. Follow the Stackdriver Installation steps to see your metrics on Stackdriver Monitoring website.

Metrics available

Name	Description	Type
agones_gameservers_count	The number of gameservers per fleet and status	gauge
agones_gameserver_allocations_duration_seconds	The distribution of gameserver allocation requests latencies	histogram
agones_gameservers_total	The total of gameservers per fleet and status	counter
agones_fleets_replicas_count	The number of replicas per fleet (total, desired, ready, allocated)	gauge
agones_fleet_autoscalers_able_to_scale	The fleet autoscaler can access the fleet to scale	gauge
agones_fleet_autoscalers_buffer_limits	The limits of buffer based fleet autoscalers (min, max)	gauge
agones_fleet_autoscalers_buffer_size	The buffer size of fleet autoscalers (count or percentage)	gauge
agones_fleet_autoscalers_current_replicas_count	The current replicas count as seen by autoscalers	gauge
agones_fleet_autoscalers_desired_replicas_count	The desired replicas count as seen by autoscalers	gauge
agones_fleet_autoscalers_limited	The fleet autoscaler is capped (1)	gauge
agones_gameservers_node_count	The distribution of gameservers per node	histogram
agones_nodes_count	The count of nodes empty and with gameservers	gauge
agones_k8s_client_http_request_total	The total of HTTP requests to the Kubernetes API by status code	counter
agones_k8s_client_http_request_duration_seconds	The distribution of HTTP requests latencies to the Kubernetes API by status code	histogram
agones_k8s_client_cache_list_total	The total number of list operations for client-go caches	counter
agones_k8s_client_cache_list_duration_seconds	Duration of a Kubernetes list API call in seconds	histogram
agones_k8s_client_cache_list_items	Count of items in a list from the Kubernetes API	histogram
agones_k8s_client_cache_watches_total	The total number of watch operations for client-go caches	counter
agones_k8s_client_cache_last_resource_version	Last resource version from the Kubernetes API	gauge
agones_k8s_client_workqueue_depth	Current depth of the work queue	gauge
agones_k8s_client_workqueue_latency_seconds	How long an item stays in the work queue	histogram
agones_k8s_client_workqueue_items_total	Total number of items added to the work queue	counter
agones_k8s_client_workqueue_work_duration_seconds	How long processing an item from the work queue takes	histogram
agones_k8s_client_workqueue_retries_total	Total number of items retried to the work queue	counter
agones_k8s_client_workqueue_longest_running_processor	How long the longest running workqueue processor has been running in microseconds	gauge
agones_k8s_client_workqueue_unfinished_work_seconds	How long unfinished work has been sitting in the workqueue in seconds	gauge

Dashboard

Grafana Dashboards

We provide a set of useful Grafana dashboards to monitor Agones workload, they are located under the grafana folder :

Agones Autoscalers allows you to monitor your current autoscalers replicas request as well as fleet replicas allocation and readyness statuses. You can only select one autoscaler at the time using the provided dropdown.
Agones GameServers displays your current game servers workload status (allocations, game servers statuses, fleets replicas) with optional fleet name filtering.
Agones GameServer Allocations displays Agones gameservers allocations rates and counts per fleet.
Agones Allocator Resource displays Agones Allocators CPU, memory usage and also some useful Golang runtime metrics.
Agones Status displays Agones controller health status.
Agones Controller Resource Usage displays Agones Controller CPU and memory usage and also some Golang runtime metrics.
Agones Controller go-client requests displays Agones Controller Kubernetes API consumption.
Agones Controller go-client caches displays Agones Controller Kubernetes Watches/Lists operations used.
Agones Controller go-client workqueues displays Agones Controller workqueue processing time and rates.
Agones Controller API Server requests displays your current API server request rate, errors rate and request latencies with optional CustomResourceDefinition filtering by Types: fleets, gameserversets, gameservers, gameserverallocations.

Dashboard screenshots :

Note

You can import our dashboards by copying the json content from each config map into your own instance of Grafana (+ > Create > Import > Or paste json) or follow the installation guide.

Installation

When operating a live multiplayer game you will need to observe performances, resource usage and availability to learn more about your system. This guide will explain how you can setup Prometheus and Grafana into your own Kubernetes cluster to monitor your Agones workload.

Before attemping this guide you should make sure you have kubectl and helm installed and configured to reach your kubernetes cluster.

Prometheus installation

Prometheus is an open source monitoring solution, we will use it to store Agones controller metrics and query back the data.

Let’s install Prometheus using the helm stable repository.

helm upgrade --install --wait prom stable/prometheus --namespace metrics \
    --set server.global.scrape_interval=30s \
    --set server.persistentVolume.enabled=true \
    --set server.persistentVolume.size=64Gi \
    -f ./build/prometheus.yaml

Note

You can also run our Makefile target make setup-prometheus or make kind-setup-prometheus and make minikube-setup-prometheus for Kind and Minikube .

For resiliency it is recommended to run Prometheus on a dedicated node which is separate from nodes where Game Servers are scheduled. If you use the above command, with our prometheus.yaml to set up Prometheus, it will schedule Prometheus pods on nodes tainted with agones.dev/agones-metrics=true:NoExecute and labeled with agones.dev/agones-metrics=true if available.

As an example, to set up a dedicated node pool for Prometheus on GKE, run the following command before installing Prometheus. Alternatively you can taint and label nodes manually.

gcloud container node-pools create agones-metrics --cluster=... --zone=... \
  --node-taints agones.dev/agones-metrics=true:NoExecute \
  --node-labels agones.dev/agones-metrics=true \
  --num-nodes=1

By default we will disable the push gateway (we don’t need it for Agones) and other exporters.

The helm chart support nodeSelector, affinity and toleration, you can use them to schedule prometheus deployments on an isolated node(s) to have an homogeneous game servers workload.

This will install a Prometheus Server in your current cluster with Persistent Volume Claim (Deactivated for Minikube and Kind) for storing and querying time series, it will automatically start collecting metrics from Agones Controller.

Finally to access Prometheus metrics, rules and alerts explorer use

kubectl port-forward deployments/prom-prometheus-server 9090 -n metrics

Note

Again you can use our Makefile make prometheus-portforward . (For Kind and Minikube use their specific targets make kind-prometheus-portforward and make minikube-prometheus-portforward)

Now you can access the prometheus dashboard http://localhost:9090.

On the landing page you can start exploring metrics by creating queries. You can also verify what targets Prometheus currently monitors (Header Status > Targets), you should see Agones controller pod in the kubernetes-pods section.

Note

Metrics will be first registered when you will start using Agones.

Now let’s install some Grafana dashboards.

Grafana installation

Grafana is a open source time series analytics platform which supports Prometheus data source. We can also easily import pre-built dashboards.

First we will install Agones dashboard as config maps in our cluster.

kubectl apply -f ./build/grafana/

Now we can install grafana chart from stable repository. (Replace <your-admin-password> with the admin password of your choice)

helm install --wait --name grafana stable/grafana --version=5.0.13 --namespace metrics \
  --set adminPassword=<your-admin-password> -f ./build/grafana.yaml

This will install Grafana with our prepopulated dashboards and prometheus datasource previously installed

Note

You can also use our Makefile targets (setup-grafana, minikube-setup-grafana and kind-setup-grafana).

Finally to access dashboards run

kubectl port-forward deployments/grafana 3000 -n metrics

Open a web browser to http://localhost:3000, you should see Agones dashboards after login as admin.

Note

You can also use our Makefile targets make grafana-portforward, make kind-grafana-portforward and make minikube-grafana-portforward.

Stackdriver installation

In order to use Stackdriver monitoring you should enable Stackdriver Monitoring API on Google Cloud Console. You need to grant all the necessary permissions to the users (see Access Control Guide). Stackdriver exporter uses a strategy called Application Default Credentials (ADC) to find your application’s credentials. Details could be found here Setting Up Authentication for Server to Server Production Applications.

Note that Stackdriver monitoring is enabled by default on GKE clusters, however you can follow this guide if it was disabled on your GKE cluster.

Default metrics exporter is Prometheus. If you are using the Helm installation, you can install or upgrade Agones to use Stackdriver, using the following chart parameters:

helm upgrade --install --wait --set agones.metrics.stackdriverEnabled=true --set agones.metrics.prometheusEnabled=false --set agones.metrics.prometheusServiceDiscovery=false my-release-name agones/agones --namespace=agones-system

With this configuration only Stackdriver exporter would be used instead of Prometheus exporter.

Create a Fleet or a Gameserver in order to check that connection with stackdriver API is configured properly and so that you will be able to see the metrics data.

Visit Stackdriver monitoring website, select your project, or choose Create a new Workspace and select GCP project where your cluster resides. In Stackdriver metrics explorer you should be able to find new metrics with prefix agones/ after a couple of minutes. Choose the metrics you are interested in and add to a single or separate graphs. Select Kubernetes Container resource type for each of them. You can create multiple graphs, save them into your dashboard and use various aggregation parameters and reducers for each graph.

Example of the dashboard appearance is provided below:

Currently there exists only manual way of configuring Stackdriver Dashboard. So it is up to you to set an Alignment Period (minimal is 1 minute), GroupBy, Filter parameters and other graph settings.

Troubleshooting

If you can’t see Agones metrics you should have a look at the controller logs for connection errors. Also ensure that your cluster has the necessary credentials to interact with Stackdriver Monitoring. You can configure stackdriverProjectID manually, if the automatic discovery is not working.

Permissions problem example from controller logs:

Failed to export to Stackdriver: rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).

Docs: Access Agones via the Kubernetes API

Thu, 03 Jan 2019 01:20:41 +0000

Installing Agones creates several Custom Resource Definitions (CRD), which can be accessed and manipulated through the Kubernetes API.

The detailed list of Agones CRDs with their parameters could be found here - Agones CRD API Reference.

Kubernetes has multiple client libraries, however, at time of writing, only the Go and Python clients are documented to support accessing CRDs.

This can be found in the Accessing a custom resource section of the Kubernetes documentation.

At this time, we recommend interacting with Agones through the Go client that has been generated in this repository, but other methods may also work as well.

Go Client

Kubernetes Go Client tooling generates a Client for Agones that we can use to interact with the Agones installation on our Kubernetes cluster.

Authentication

This client uses the same authentication mechanisms as the Kubernetes API.

If you plan to run your code in the same cluster as the Agones install, have a look at the in cluster configuration example from the Kubernetes Client.

If you plan to run your code outside the Kubernetes cluster as your Agones install, look at the out of cluster configuration example from the Kubernetes client.

Example

The following is an example of a in-cluster configuration, that creates a Clientset for Agones and then creates a GameServer.

A full example code is available in the example folder .

package main

import (
	"fmt"

	agonesv1 "agones.dev/agones/pkg/apis/agones/v1"
	"agones.dev/agones/pkg/client/clientset/versioned"
	"agones.dev/agones/pkg/util/runtime"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
)

func main() {
	config, err := rest.InClusterConfig()
	logger := runtime.NewLoggerWithSource("main")
	if err != nil {
		logger.WithError(err).Fatal("Could not create in cluster config")
	}

	// Access to standard Kubernetes resources through the Kubernetes Clientset
	// We don't actually need this for this example, but it's just here for
	// illustrative purposes
	kubeClient, err := kubernetes.NewForConfig(config)
	if err != nil {
		logger.WithError(err).Fatal("Could not create the kubernetes clientset")
	}

	// Access to the Agones resources through the Agones Clientset
	// Note that we reuse the same config as we used for the Kubernetes Clientset
	agonesClient, err := versioned.NewForConfig(config)
	if err != nil {
		logger.WithError(err).Fatal("Could not create the agones api clientset")
	}

	// Create a GameServer
	gs := &agonesv1.GameServer{ObjectMeta: metav1.ObjectMeta{GenerateName: "udp-server", Namespace: "default"},
		Spec: agonesv1.GameServerSpec{
			Container: "udp-server",
			Ports: []agonesv1.GameServerPort{{
				ContainerPort: 7654,
				HostPort:      7654,
				Name:          "gameport",
				PortPolicy:    agonesv1.Static,
				Protocol:      corev1.ProtocolUDP,
			}},
			Template: corev1.PodTemplateSpec{
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{{Name: "udp-server", Image: "gcr.io/agones-images/udp-server:0.21"}},
				},
			},
		},
	}
	newGS, err := agonesClient.AgonesV1().GameServers("default").Create(gs)
	if err != nil {
		panic(err)
	}

	fmt.Printf("New game servers' name is: %s", newGS.ObjectMeta.Name)
}

In order to create GS using provided example, you can run it as a Kubernetes Job:

$ kubectl create -f https://raw.githubusercontent.com/googleforgames/agones/master/examples/crd-client/create-gs.yaml --namespace agones-system
$ kubectl get pods --namespace agones-system
NAME                                 READY   STATUS      RESTARTS   AGE
create-gs-6wz86-7qsm5                0/1     Completed   0          6s
$ kubectl logs create-gs-6wz86-7qsm5  --namespace agones-system
{"message":"\u0026{0xc0001dde00 default}","severity":"info","source":"main","time":"2020-04-21T11:14:00.477576428Z"}
{"message":"New GameServer name is: helm-test-server-fxfgg","severity":"info","time":"2020-04-21T11:14:00.516024697Z"}

You have just created a GameServer using Kubernetes Go Client.

Direct Access to the REST API via Kubectl

If there isn’t a client written in your preferred language, it is always possible to communicate directly with Kubernetes API to interact with Agones.

The Kubernetes API can be authenticated and exposed locally through the kubectl proxy

For example:

$ kubectl proxy &
Starting to serve on 127.0.0.1:8001

# list all Agones endpoints
$ curl http://localhost:8001/apis | grep agones -A 5 -B 5
...
    {
      "name": "agones.dev",
      "versions": [
        {
          "groupVersion": "agones.dev/v1",
          "version": "v1"
        }
      ],
      "preferredVersion": {
        "groupVersion": "agones.dev/v1",
        "version": "v1"
      },
      "serverAddressByClientCIDRs": null
    }
...

# List Agones resources
$ curl http://localhost:8001/apis/agones.dev/v1
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "agones.dev/v1",
  "resources": [
    {
      "name": "gameservers",
      "singularName": "gameserver",
      "namespaced": true,
      "kind": "GameServer",
      "verbs": [
        "delete",
        "deletecollection",
        "get",
        "list",
        "patch",
        "create",
        "update",
        "watch"
      ],
      "shortNames": [
        "gs"
      ]
    }
  ]
}

# list all gameservers in the default namespace
$ curl http://localhost:8001/apis/agones.dev/v1/namespaces/default/gameservers
{
    "apiVersion": "agones.dev/v1",
    "items": [
        {
            "apiVersion": "agones.dev/v1",
            "kind": "GameServer",
            "metadata": {
                "annotations": {
                    "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"agones.dev/v1\",\"kind\":\"GameServer\",\"metadata\":{\"annotations\":{},\"name\":\"simple-udp\",\"namespace\":\"default\"},\"spec\":{\"containerPort\":7654,\"hostPort\":7777,\"portPolicy\":\"static\",\"template\":{\"spec\":{\"containers\":[{\"image\":\"gcr.io/agones-images/udp-server:0.21\",\"name\":\"simple-udp\"}]}}}}\n"
                },
                "clusterName": "",
                "creationTimestamp": "2018-03-02T21:41:05Z",
                "finalizers": [
                    "agones.dev"
                ],
                "generation": 0,
                "name": "simple-udp",
                "namespace": "default",
                "resourceVersion": "760",
                "selfLink": "/apis/agones.dev/v1/namespaces/default/gameservers/simple-udp",
                "uid": "692beea6-1e62-11e8-beb2-080027637781"
            },
            "spec": {
                "PortPolicy": "Static",
                "container": "simple-udp",
                "containerPort": 7654,
                "health": {
                    "failureThreshold": 3,
                    "initialDelaySeconds": 5,
                    "periodSeconds": 5
                },
                "hostPort": 7777,
                "protocol": "UDP",
                "template": {
                    "metadata": {
                        "creationTimestamp": null
                    },
                    "spec": {
                        "containers": [
                            {
                                "image": "gcr.io/agones-images/udp-server:0.21",
                                "name": "simple-udp",
                                "resources": {}
                            }
                        ]
                    }
                }
            },
            "status": {
                "address": "192.168.99.100",
                "nodeName": "agones",
                "port": 7777,
                "state": "Ready"
            }
        }
    ],
    "kind": "GameServerList",
    "metadata": {
        "continue": "",
        "resourceVersion": "1062",
        "selfLink": "/apis/agones.dev/v1/namespaces/default/gameservers"
    }
}

# allocate a gameserver from a fleet named 'simple-udp', with GameServerAllocation

$ curl -d '{"apiVersion":"allocation.agones.dev/v1","kind":"GameServerAllocation","spec":{"required":{"matchLabels":{"agones.dev/fleet":"simple-udp"}}}}' -H "Content-Type: application/json" -X POST http://localhost:8001/apis/allocation.agones.dev/v1/namespaces/default/gameserverallocations

{
    "kind": "GameServerAllocation",
    "apiVersion": "allocation.agones.dev/v1",
    "metadata": {
        "name": "simple-udp-v6jwb-cmdcv",
        "namespace": "default",
        "creationTimestamp": "2019-07-03T17:19:47Z"
    },
    "spec": {
        "multiClusterSetting": {
            "policySelector": {}
        },
        "required": {
            "matchLabels": {
                "agones.dev/fleet": "simple-udp"
            }
        },
        "scheduling": "Packed",
        "metadata": {}
    },
    "status": {
        "state": "Allocated",
        "gameServerName": "simple-udp-v6jwb-cmdcv",
        "ports": [
            {
                "name": "default",
                "port": 7445
            }
        ],
        "address": "34.94.118.237",
        "nodeName": "gke-test-cluster-default-f11755a7-5km3"
    }
}

You may wish to review the Agones Kubernetes API for the full data structure reference.

The Kubernetes API Concepts section may also provide the more details on the API conventions that are used in the Kubernetes API.

Next Steps

Learn how to use Allocator Service for single and multi-cluster Allocation.

Docs: Troubleshooting

Thu, 03 Jan 2019 01:20:49 +0000

Something went wrong with my GameServer

If there is something going wrong with your GameServer, there are a few approaches to determining the cause:

Depending on what is happening, you may want to run kubectl describe <gameserver name> to view the events that are associated with that particular GameServer resource. This can give you insight into the lifecycle of the GameServer and if anything has gone wrong.

For example, here we can see where the simple-udp example has been moved to the Unhealthy state due to a crash in the backing GameServer Pod container’s binary.

root@6a71afd42291:/go/src/agones.dev/agones# kubectl describe gs simple-udp-zqppv
Name:         simple-udp-zqppv
Namespace:    default
Labels:       <none>
Annotations:  agones.dev/sdk-version: 1.0.0-dce1546
API Version:  agones.dev/v1
Kind:         GameServer
Metadata:
  Creation Timestamp:  2019-08-16T21:25:44Z
  Finalizers:
    agones.dev
  Generate Name:     simple-udp-
  Generation:        1
  Resource Version:  1378575
  Self Link:         /apis/agones.dev/v1/namespaces/default/gameservers/simple-udp-zqppv
  UID:               6818adc7-c06c-11e9-8dbd-42010a8a0109
Spec:
  Container:  simple-udp
  Health:
    Failure Threshold:      3
    Initial Delay Seconds:  5
    Period Seconds:         5
  Ports:
    Container Port:  7654
    Host Port:       7058
    Name:            default
    Port Policy:     Dynamic
    Protocol:        UDP
  Scheduling:        Packed
  Template:
    Metadata:
      Creation Timestamp:  <nil>
    Spec:
      Containers:
        Image:  gcr.io/agones-images/udp-server:0.21
        Name:   simple-udp
        Resources:
          Limits:
            Cpu:     20m
            Memory:  32Mi
          Requests:
            Cpu:     20m
            Memory:  32Mi
Status:
  Address:    35.230.59.117
  Node Name:  gke-test-cluster-default-590db5e4-4s6r
  Ports:
    Name:          default
    Port:          7058
  Reserved Until:  <nil>
  State:           Unhealthy
Events:
  Type     Reason          Age   From                   Message
  ----     ------          ----  ----                   -------
  Normal   PortAllocation  72s   gameserver-controller  Port allocated
  Normal   Creating        72s   gameserver-controller  Pod simple-udp-zqppv created
  Normal   Scheduled       72s   gameserver-controller  Address and port populated
  Normal   RequestReady    67s   gameserver-sidecar     SDK state change
  Normal   Ready           66s   gameserver-controller  SDK.Ready() complete
  Warning  Unhealthy       34s   health-controller      Issue with Gameserver pod

The backing Pod has the same name as the GameServer - so it’s also worth looking at the details and events for the Pod to see if there are any issues there, such as restarts due to binary crashes etc.

For example, you can see the restart count on the gcr.io/agones-images/udp-server:0.21 container is set to 1, due to the game server binary crash

root@6a71afd42291:/go/src/agones.dev/agones# kubectl describe pod simple-udp-zqppv
Name:               simple-udp-zqppv
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-test-cluster-default-590db5e4-4s6r/10.138.0.23
Start Time:         Fri, 16 Aug 2019 21:25:44 +0000
Labels:             agones.dev/gameserver=simple-udp-zqppv
                    agones.dev/role=gameserver
Annotations:        agones.dev/container: simple-udp
                    agones.dev/sdk-version: 1.0.0-dce1546
                    cluster-autoscaler.kubernetes.io/safe-to-evict: false
Status:             Running
IP:                 10.48.1.80
Controlled By:      GameServer/simple-udp-zqppv
Containers:
  simple-udp:
    Container ID:   docker://69eacd03cc89b0636b78abe47926b02183ba84d18fa20649ca443f5232511661
    Image:          gcr.io/agones-images/udp-server:0.21
    Image ID:       docker-pullable://gcr.io/agones-images/udp-server@sha256:6a60eff5e68b88b5ce75ae98082d79cff36cda411a090f3495760e5c3b6c3575
    Port:           7654/UDP
    Host Port:      7058/UDP
    State:          Running
      Started:      Fri, 16 Aug 2019 21:26:22 +0000
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 16 Aug 2019 21:25:45 +0000
      Finished:     Fri, 16 Aug 2019 21:26:22 +0000
    Ready:          True
    Restart Count:  1
    Limits:
      cpu:     20m
      memory:  32Mi
    Requests:
      cpu:        20m
      memory:     32Mi
    Liveness:     http-get http://:8080/gshealthz delay=5s timeout=1s period=5s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from empty (ro)
  agones-gameserver-sidecar:
    Container ID:   docker://f3c475c34d26232e19b60be65b03bc6ce41931f4c37e00770d3ab4a36281d31c
    Image:          gcr.io/agones-mark/agones-sdk:1.0.0-dce1546
    Image ID:       docker-pullable://gcr.io/agones-mark/agones-sdk@sha256:4b5693e95ee3023a2b2e2099d102bb6bac58d4ce0ac472e58a09cee6d160cd19
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 16 Aug 2019 21:25:48 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     30m
    Liveness:  http-get http://:8080/healthz delay=3s timeout=1s period=3s #success=1 #failure=3
    Environment:
      GAMESERVER_NAME:  simple-udp-zqppv
      POD_NAMESPACE:    default (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from agones-sdk-token-vr6qq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  empty:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  agones-sdk-token-vr6qq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  agones-sdk-token-vr6qq
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age                   From                                             Message
  ----    ------     ----                  ----                                             -------
  Normal  Scheduled  2m32s                 default-scheduler                                Successfully assigned default/simple-udp-zqppv to gke-test-cluster-default-590db5e4-4s6r
  Normal  Pulling    2m31s                 kubelet, gke-test-cluster-default-590db5e4-4s6r  pulling image "gcr.io/agones-mark/agones-sdk:1.0.0-dce1546"
  Normal  Started    2m28s                 kubelet, gke-test-cluster-default-590db5e4-4s6r  Started container
  Normal  Pulled     2m28s                 kubelet, gke-test-cluster-default-590db5e4-4s6r  Successfully pulled image "gcr.io/agones-mark/agones-sdk:1.0.0-dce1546"
  Normal  Created    2m28s                 kubelet, gke-test-cluster-default-590db5e4-4s6r  Created container
  Normal  Created    114s (x2 over 2m31s)  kubelet, gke-test-cluster-default-590db5e4-4s6r  Created container
  Normal  Started    114s (x2 over 2m31s)  kubelet, gke-test-cluster-default-590db5e4-4s6r  Started container
  Normal  Pulled     114s (x2 over 2m31s)  kubelet, gke-test-cluster-default-590db5e4-4s6r  Container image "gcr.io/agones-images/udp-server:0.21" already present on machine

Finally, you can also get the logs of your GameServer Pod as well via kubectl logs <pod name> -c <game server container name>, for example:

root@6a71afd42291:/go/src/agones.dev/agones# kubectl logs simple-udp-zqppv -c simple-udp
2019/08/16 21:26:23 Creating SDK instance
2019/08/16 21:26:24 Starting Health Ping
2019/08/16 21:26:24 Starting UDP server, listening on port 7654
2019/08/16 21:26:24 Marking this server as ready

The above commands will only give the most recent container’s logs (so we won’t get the previous crash), but you can use kubectl logs --previous=true simple-udp-zqppv -c simple-udp to get the previous instance of the containers logs, or use your Kubernetes platform of choice’s logging aggregation tools to view the crash details.

For more tips and tricks, the Kubernetes Cheatsheet: Interactive with Pods also provides more troubleshooting techniques.

How do I see the logs for Agones?

If something is going wrong, and you want to see the logs for Agones, there are potentially two places you will want to check:

The controller: assuming you installed Agones in the agones-system namespace, you will find that there is a single pod called agones-controller-<hash> (where hash is the unique code that Kubernetes generates) that exists there, that you can get the logs from. This is the main controller for Agones, and should be the first place to check when things go wrong.
1. To get the logs from this controller run:
  kubectl logs --namespace=agones-system agones-controller-<hash>
The SDK server sidecar: Agones runs a small gRPC + http server for the SDK in a container in the same network namespace as the game server container to connect to via the SDK.
The logs from this SDK server are also useful for tracking down issues, especially if you are having trouble with a particular GameServer.
1. To find the Pod for the GameServer look for the pod with a name that is prefixed with the name of the owning GameServer. For example if you have a GameServer named simple-udp, it’s pod could potentially be named simple-udp-dnbwj.
2. To get the logs from that Pod, we need to specify that we want the logs from the agones-gameserver-sidecar container. To do that, run the following:
  kubectl logs simple-udp-dnbwj -c agones-gameserver-sidecar

Agones uses JSON structured logging, therefore errors will be visible through the "severity":"info" key and value.

I uninstalled Agones before deleted all my `GameServers` and now they won’t delete

Agones GameServers use Finalizers to manage garbage collection of the GameServers. This means that if the Agones controller doesn’t remove the finalizer for you (i.e. if it has been uninstalled), it can be tricky to remove them all.

Thankfully, if we create a patch to remove the finalizers from all GameServers, we can delete them with impunity.

A quick one liner to do this:

kubectl get gameserver -o name | xargs -n1 -P1 -I{} kubectl patch {} --type=merge -p '{"metadata": {"finalizers": []}}'

Once this is done, you can kubectl delete gs --all and clean everything up (if it’s not gone already).

I’m getting Forbidden errors when trying to install Agones

Ensure that you are running Kubernetes 1.12 or later, which does not require any special clusterrolebindings to install Agones.

If you want to install Agones on an older version of Kubernetes, you need to create a clusterrolebinding to add your identity as a cluster admin, e.g.

# Kubernetes Engine
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole cluster-admin --user `gcloud config get-value account`
# Minikube
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin --serviceaccount=kube-system:default

On GKE, gcloud config get-value accounts will return a lowercase email address, so if you are using a CamelCase email, you may need to type it in manually.