Cloud Logging for Storage Transfer Service

gcloud CLI

When creating a transfer job with gcloud transfer jobs create, use the following flags to enable logging:

gcloud transfer jobs create SOURCE DESTINATION \
  --log-actions=copy,delete,find \
  --log-action-states=succeeded,failed

You must specify at least one value for each flag.

REST

To create a logging configuration, use transferJobs.create with a LoggingConfig:

{
  "name":"transferJobs/myFirstTransfer",
  "status": "ENABLED",
  "projectId": "test-id-001",
  "loggingConfig": {
     "logActions": ["FIND", "DELETE", "COPY"],
     "logActionStates": ["SUCCEEDED", "FAILED"],
  },
  "transferSpec": {
      "awsS3DataSource": {
          "bucketName": "AWS_SOURCE_NAME",
          "awsAccessKey": {
              "accessKeyId": "AWS_ACCESS_KEY_ID",
              "secretAccessKey": "AWS_SECRET_ACCESS_KEY"
          }
      },
      "gcsDataSink": {
           "bucketName": "destination_bucket",
           "path": "foo/bar/"
      },
   }
}

Adjust loggingConfig to include the specific logActions and logActionStates to log. For example, to log when copy and find actions fail, provide the following loggingConfig:

"loggingConfig": {
  "logActions": ["COPY", "FIND"],
  "logActionStates": ["FAILED"],
}

gcloud CLI

To update an existing job's logging configuration, use the appropriate flags with the gcloud transfer jobs update command:

gcloud transfer jobs update NAME \
  --log-actions=copy,delete,find \
  --log-action-states=succeeded,failed

To disable logging for this job, specify --clear-log-config:

gcloud transfer jobs update NAME --clear-log-config

REST

To update an existing transfer job's logging configuration, use transferJobs.patch with LoggingConfig:

{
  "projectId": "test-id-001",
  "transferJob": {
    "loggingConfig": {
       "logActions": ["FIND", "DELETE", "COPY"],
       "logActionStates": ["SUCCEEDED", "FAILED"],
    },
  },
  "updateTransferJobFieldMask": "loggingConfig"
}

The updateTransferJobFieldMask specifies the field that is being updated in this request and is required.

To disable logging for this job, send a loggingConfig with empty lists for logActions and logActionStates:

{
  "projectId": "test-id-001",
  "transferJob": {
    "loggingConfig": {
       "logActions": [],
       "logActionStates": [],
    },
  },
  "updateTransferJobFieldMask": "loggingConfig"
}

Google Cloud console

1. Go to the Google Cloud navigation menu menu and select Logging > Logs Explorer :

   Go to the Logs Explorer

2. Select a Google Cloud project.

3. From the Upgrade menu, switch from Legacy Logs Viewer to Logs Explorer.

4. To filter your logs to show only Storage Transfer Service entries, type storage_transfer_job into the query field and click Run query.

5. In the Query results pane, click Edit time to change the time period for which to return results.

For more information on using the Logs Explorer, see Using the Logs Explorer.

gcloud CLI

To use the gcloud CLI to search for Storage Transfer Service logs, use the gcloud logging read command.

Specify a filter to limit your results to Storage Transfer Service logs.

gcloud logging read "resource.type=storage_transfer_job"

Cloud Logging API

Use the entries.list Cloud Logging API method.

To filter your results to include only Storage Transfer Service-related entries, use the filter field. A sample JSON request object is below.

{
"resourceNames":
  [
    "projects/my-project-name"
  ],
  "orderBy": "timestamp desc",
  "filter": "resource.type=\"storage_transfer_job\""
}

FIND actions

{
  "jsonPayload": {
    "@type": "type.googleapis.com/google.storagetransfer.logging.TransferActivityLog",
    "action": "FIND",
    "completeTime": "2021-12-16T18:58:49.344509695Z",
    "destinationContainer": {
      "gcsBucket": {
        "bucket": "my-bucket-2",
      },
      "type": "GCS",
    },
    "operation": "transferOperations/transferJobs-7876027868280507149--3019866490856027148",
    "sourceContainer": {
      "gcsBucket": {
        "bucket": "my-bucket-1"
      },
      "type": "GCS"
    },
    "status": {
      "statusCode": "OK"
    }
  }
}

COPY and DELETE actions

{
  "jsonPayload": {
    "@type": "type.googleapis.com/google.storagetransfer.logging.TransferActivityLog",
    "action": "COPY",
    "completeTime": "2021-12-16T18:59:00.510509049Z",
    "destinationObject": {
      "gcsObject": {
        "bucket": "my-bucket-2",
        "objectKey": "README.md"
      },
      "type": "GCS",
    },
    "operation": "transferOperations/transferJobs-7876027868280507149--3019866490856027148",
    "sourceObject": {
      "gcsObject": {
        "bucket": "my-bucket-1",
        "lastModifiedTime": "2021-12-07T16:41:09.456Z",
        "md5": "WgnCOIdfCXNTUDpQJSKb2w==",
        "objectKey": "README.md",
      },
      "type": "GCS",
    },
    "status": {
      "statusCode": "OK"
    }
  }
}