Description
Environments
- fluentd version: v1.15.0-1.0
- plugin version: 3.0.0
Configuration
<match audit-logs-bigquery.**>
@type bigquery_load
<buffer>
path /mnt/audit-logs-bigquery.*.buffer
flush_interval 10s
total_limit_size 1g
</buffer>
auth_method json_key
json_key "#{ENV['FLUENT_BIGQUERY_JSON_KEY']}"
project "#{ENV['FLUENT_BIGQUERY_PROJECT']}"
dataset "#{ENV['FLUENT_BIGQUERY_DATASET']}"
table "#{ENV['FLUENT_BIGQUERY_TABLE']}"
@log_level "{'info'}"
</match>
Expected Behavior
No Error
Actual Behavior
I believe all the logs are being written as expected, but I am seeing this error many times, I am not sure I understand what the error is and was wondering if there is something wrong on my end, or with the plugin. I found this old issue here, but its with the insert and I need to use the load as the insert does not support the JSON data type.
Log (if you have)
2022-10-15 19:07:57.816134286 +0000 fluent.error: {"error":"#<Fluent::BigQuery::UnRetryableError: notFound: Not found: Job project-id-01:job_GrVSui91crPvhn2DqmBvVCc2upYA>","chunk":"5eb1774d7fd7dc5d0c6f454e4d9a7839","project_id":"project-id-01","dataset_id":"dataset","table_id":"table_name","job_id":"job_GrVSui91crPvhn2DqmBvVCc2upYA","message":"failed to poll load job error_class=Fluent::BigQuery::UnRetryableError error=\"notFound: Not found: Job project-id-01:job_GrVSui91crPvhn2DqmBvVCc2upYA\" chunk=\"5eb1774d7fd7dc5d0c6f454e4d9a7839\" project_id=\"project-id-01\" dataset_id=\"dataset\" table_id=\"table_name\" job_id=\"job_GrVSui91crPvhn2DqmBvVCc2upYA\""}