CLI 调用

您可以使用 Google Cloud CLI 编写 Dataflow 作业的执行脚本。通过 CLI 执行作业在功能上等同于使用 Dataflow 用户界面。

export REGION=us-central1
gcloud dataflow flex-template run "test-bq-cli-`date +%Y%m%d-%H%M%S`" \
  --template-file-gcs-location="gs://dataflow-templates/latest/flex/Google_Cloud_to_Neo4j" \
  --region "$REGION" \
  --parameters jobSpecUri="<URI-to-job-specification-JSON-file>" \
  --parameters neo4jConnectionUri="<URI-to-neo4j-connection-JSON-file>"

如果凭据文件存储在 Google Secret Manager 中，请将参数 neo4jConnectionUri 替换为 neo4jConnectionSecretId。调用变为

export REGION=us-central1
gcloud dataflow flex-template run "test-bq-cli-`date +%Y%m%d-%H%M%S`" \
  --template-file-gcs-location="gs://dataflow-templates/latest/flex/Google_Cloud_to_Neo4j" \
  --region "$REGION" \
  --parameters jobSpecUri="<URI-to-job-specification-JSON-file>" \
  --parameters neo4jConnectionSecretId="<secret-id>"

其中 <secret-id> 具有以下结构：projects/<project>/secrets/<secret-name>/versions/<secret-version>。

REST 版本如下所示

export REGION=us-central1
curl -X POST "https://Dataflow.googleapis.com/v1b3/projects/<project-name>/locations/$REGION/flexTemplates:launch" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-d '{
   "launch_parameter": {
      "jobName": "test-bq-rest-'$(date +%Y%m%d-%H%M%S)'",
      "parameters": {
        "jobSpecUri": "<URI-to-job-specification-JSON-file>",
        "neo4jConnectionUri": "<URI-to-neo4j-connection-JSON-file>"
      },
   "containerSpecGcsPath": "gs://dataflow-templates/latest/flex/Google_Cloud_to_Neo4j"
   }
}'

如果凭据文件存储在 Google Secret Manager 中，请将参数 neo4jConnectionUri 替换为 neo4jConnectionSecretId。调用变为

export REGION=us-central1
curl -X POST "https://Dataflow.googleapis.com/v1b3/projects/<project-name>/locations/$REGION/flexTemplates:launch" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-d '{
   "launch_parameter": {
      "jobName": "test-bq-rest-'$(date +%Y%m%d-%H%M%S)'",
      "parameters": {
        "jobSpecUri": "<URI-to-job-specification-JSON-file>",
        "neo4jConnectionSecretId": "<secret-id>"
      },
   "containerSpecGcsPath": "gs://dataflow-templates/latest/flex/Google_Cloud_to_Neo4j"
   }
}'

Neo4j Dataflow 模板存储库中的 cli-script 文件夹包含一些示例调用。