Exporting Logs in the API  |  Stackdriver Logging  |  Google Cloud

This guide explains how to use the Stackdriver Logging API to export your logs out
of Stackdriver Logging. To instead use the Logs Viewer to set up exports,
see Logs Viewer: Exporting Logs. To
use the command-line SDK to set up exports,
see SDK: Exporting Logs. To
find and use exported logs,
see Using Exported Logs.

Introduction to sinks

You control the export of log entries by creating sinks in
Stackdriver Logging. Sinks contain the following information:

  • A sink identifier that names the sink within the parent. Examples

    “my-first-sink”

  • A parent resource—a project, folder, billing account, or
    organization where in which sink is created. The sink can only export
    logs belonging to the parent resource.
    The parents have the following formats:

    “projects/[PROJECT_ID]”
    “organizations/[ORGANIZATION_ID]”
    “folders/[FOLDER_ID]”
    “billingAccounts/[BILLING_ACCOUNT_ID]”

  • An include children parameter. This is only relevant to sinks within
    organizations. See
    Aggregated Exports.

  • A logs filter to select which log entries to export.
    For more information, see Advanced Logs Filters. The following
    example matches all log entries with severity WARNING or higher from
    a specified Compute Engine VM instance:

    “resource.type = gce_instance AND
    resource.labels.instance_id = “1428064241541024269” AND
    severity >= WARNING”

  • A destination to which the log entries are exported. This can be
    a Cloud Storage bucket, a BigQuery dataset, or a Cloud Pub/Sub topic,
    all of which must belong to a GCP project.
    The destinations have the following format:

    “storage.googleapis.com/[BUCKET_ID]”
    “bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]”
    “pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_ID]”

    You can export the log
    entries to a destination in any GCP project. The
    sink’s source and the destination do not have to be within the same
    organization.

    It is not required that you have writer access to the export
    destination. Only the sink’s writer service account,
    explained next, requires access.

  • A writer identity that is a service account name. The destination’s
    owner must give this service account permission to write to the destination.
    When exporting logs, Stackdriver Logging adopts this identity for
    authorization. For increased security, new sinks get their own unique
    service account:

    [GENERATED_ID_1]@[GENERATED_ID_2].iam.gserviceaccount.com

    Older sinks might use a shared service account for this writer:

    cloud-logs@system.gserviceaccount.com

    For more information on managing the writer identity,
    see Destination permissions.

For more information on sinks, see the LogSink type and the
projects.sinks.create API method.

How sinks work

When a log entry arrives in a project, folder, billing account, or organization,
Stackdriver Logging compares the log entry with the filters in the sinks
created in that parent. Each sink whose filter matches the log entry writes a
copy of the log entry to the sink’s destination.

Organization and folder sinks can optionally be permitted to export log entries
from their projects.
See Aggregated Exports.

Caution: A sink can export any kind of log entry, including Admin Activity audit
logs and private Data Access audit logs. For more information,
see Access control on this page.

Troubleshooting: A sink might not export matching log entries for the
following reasons:

  • The sink’s writer identity does not have permission to write logs to the
    destination. Errors occur until the sink’s writer has permission to write
    log entries to the destination.

For information about delays in finding exported logs at your destination, see
Using exported logs.

Creating sinks

This section explains how to create and modify sinks. Sinks can only export
logs from the same resource that contains the sink—a particular project,
folder, billing account, or organization.

Using a unique writer identity

You can use either a unique writer identity (a generated service account) or the
backward-compatible shared writer identity (a common group) for your sink. For
better security, you should prefer to use a unique writer for all new sinks.
With a unique writer, the destination administrator knows that granting write
permission to the identity will not allow any other sinks to write to the
destination.

You must use a unique writer if your sink’s parent is not a project
or if you are exporting to a destination in a different project than the one
containing the sink. The shared writer works only when the destination
and the sink are in the same project.

The following steps show you how to create or modify a sink:

  1. Create the sink’s destination, if needed.

  2. Call the API method projects.sinks.create or projects.sinks.update
    to create or modify the sink. The following parameters are important for
    these instructions:

    • Set uniqueWriterIdentity to true. When updating a sink, you can
      change from using a shared writer to a unique writer. If the existing
      sink already uses a unique writer, the updated sink uses the same
      writer.

      If you are updating a sink that already uses a unique writer, do not
      default the uniqueWriterIdentity parameter. The default value is
      false, which is not permitted in this case.

    The methods return the new sink, which contains the new writer identity.

  3. Add the new sink’s writer identity to the destination’s permission list,
    giving the writer permission to write to the destination.

In new sinks, using the shared writer identity is discouraged.

You can use the backward-compatible, shared writer identity only if your sink is
in the same project as your destination. Otherwise, the shared writer cannot
get permission to write to the destination.

To create or update a sink with the shared writer, do the following:

  1. Create the sink’s destination, if needed.

  2. Add one of the following writers—they are
    equivalent—to your destination’s permission list, giving the writer
    permission to write to the destination:

    • the group, cloud-logs@google.com
    • the service account ,cloud-logs@system.gserviceaccount.com
  3. Call the API method projects.sinks.create or projects.sinks.update
    to create or modify your sink. The following parameter is relevant to
    these instructions:

    • Set uniqueWriterIdentity to false. If you are updating a sink,
      the existing sink must already be using the shared writer; you cannot
      change a sink from a unique writer to a shared writer.

    The new sink begins exporting log entries immediately.

Access control

There are three issues relevant to controlling access to exported logs:

  • Permitting users or applications to create and modify sinks.
  • Permitting a sink’s writer identity to write exported logs
    to a destination.
  • Permitting users or applications to read or modify exported logs.

Managing sinks

The ability to create and view sinks is controlled
by Logging IAM roles in the sink’s parent:

  • To create or modify sinks, you or your application must have the IAM role
    Logging/Logs Configuration Writer or be an Owner of the sink’s
    parent project or other parent resource.

  • To view existing sinks, you or your application must have the IAM
    roll Logging/Logs Viewer or be a Viewer of the sink’s parent project
    or other parent resource.

Caution: All sinks can export audit logs, including private Data Access audit
logs, without extra permissions.

Writing to the destination

When Stackdriver Logging is exporting logs through a sink, it can run under
different writer identities:

Writer identity Type When creating the sink
cloud-logs@google.com group uniqueWriterIdentity=false
cloud-logs@system.gserviceaccount.com service account uniqueWriterIdentity=false
xxxxxxx@yyyyyy.gserviceaccount.com service account uniqueWriterIdentity=true

The first two writers are equivalent. The third identity is a unique writer for
each sink, which has the advantage of letting you export logs to a destination
in a project other than the sink’s parent. If the sink’s parent is not a
project, you must use a unique writer, since all destinations are in projects.

Once you choose a kind of writer, use it in a way appropriate to each kind of
destination:

  • In Cloud Storage, add the writer to your destination bucket’s
    permission list and give the writer the
    Storage Object Creator
    role.

  • In BigQuery, add the writer to your destination dataset’s permission list
    and give the writer the BigQuery Data Editor role.

  • In Cloud Pub/Sub, add the writer to your destination topic’s permission list
    and give the writer the
    Pub/Sub Publisher
    role.

Reading from the destination

It is the destination’s responsibility to control access to exported log
entries. For example, Stackdriver Logging prevents the deletion or
modification of audit logs that are held in Stackdriver Logging, and can
restrict access to private audit logs. When audit logs are exported, the
exported copies are no longer under Stackdriver Logging’s control.

Code samples

List your sinks

List the sinks in a project to keep track of where your logs are being exported.

C#

Go

Java

Node.js

PHP

Python

Ruby

Create a sink

Create a sink in your project to establish a destination to which you can
send your log entries. Before creating the sink, be sure you have granted
cloud-logs@system.gserviceaccount.com permission to write logs to the
destination. For more information, see
Permissions and Roles.

C#

Go

Java

Node.js

PHP

Python

Ruby

Update a sink

Update a sink in a project to change its destination. For example, you might
want to change the Cloud Storage bucket to which you are exporting logs monthly.

To update the sink, supply the sink’s name and a new LogSink object
containing the new destination. Before updating the sink, be sure you have
given the sink’s writer identity permission to write logs to the destination.
For more information, see
Permissions and Roles.

C#

Go

Java

Node.js

PHP

Python

Ruby

Delete a sink

Delete a sink from your project if you no longer want to export log entries
through that sink.

C#

Go

Java

Node.js

PHP

Python

Ruby

Source

Leave a comment

Your email address will not be published. Required fields are marked *