Google Exam Professional Data Engineer Topic 1 Question 101 Discussion

Actual exam question for Google's Professional Data Engineer exam

Question #: 101
Topic #: 1

[All Professional Data Engineer Questions]

Your car factory is pushing machine measurements as messages into a Pub/Sub topic in your Google Cloud project. A Dataflow streaming job. that you wrote with the Apache Beam SDK, reads these messages, sends acknowledgment lo Pub/Sub. applies some custom business logic in a Doffs instance, and writes the result to BigQuery. You want to ensure that if your business logic fails on a message, the message will be sent to a Pub/Sub topic that you want to monitor for alerting purposes. What should you do?

AUse an exception handling block in your Data Flow's Doffs code to push the messages that failed to be transformed through a side output
and to a new Pub/Sub topic. Use Cloud Monitoring to monitor the topic/num_jnacked_messages_by_region metric on this new topic.

BEnable retaining of acknowledged messages in your Pub/Sub pull subscription. Use Cloud Monitoring to monitor the
subscription/num_retained_acked_messages metric on this subscription.

CEnable dead lettering in your Pub/Sub pull subscription, and specify a new Pub/Sub topic as the dead letter topic. Use Cloud Monitoring to
monitor the subscription/dead_letter_message_count metric on your pull subscription.

DCreate a snapshot of your Pub/Sub pull subscription. Use Cloud Monitoring to monitor the snapshot/numessages metric on this
snapshot.

Show Suggested Answer

Suggested Answer: C

To ensure that messages failing to process in your Dataflow job are sent to a Pub/Sub topic for monitoring and alerting, the best approach is to use Pub/Sub's dead-letter topic feature. Here's why option C is the best choice:

Dead-Letter Topic:

Pub/Sub's dead-letter topic feature allows messages that fail to be processed successfully to be redirected to a specified topic. This ensures that these messages are not lost and can be reviewed for debugging and alerting purposes.

Monitoring and Alerting:

By specifying a new Pub/Sub topic as the dead-letter topic, you can use Cloud Monitoring to track metrics such as subscription/dead_letter_message_count, providing visibility into the number of failed messages.

This allows you to set up alerts based on these metrics to notify the appropriate teams when failures occur.

Steps to Implement:

Enable Dead-Letter Topic:

Configure your Pub/Sub pull subscription to enable dead lettering and specify the new Pub/Sub topic for dead-letter messages.

Set Up Monitoring:

Use Cloud Monitoring to monitor the subscription/dead_letter_message_count metric on your pull subscription.

Configure alerts based on this metric to notify the team of any processing failures.

Pub/Sub Dead Letter Policy

Cloud Monitoring with Pub/Sub

by Arthur at Jan 10, 2025, 12:51 PM

Limited Time Offer

25%

Off

Get Premium Professional Data Engineer Questions as Interactive Web-Based Practice Test or PDF

Contribute your Thoughts:

Submit Cancel

Letha

9 days ago

Option A seems like the way to go. I like the idea of using a side output to handle failed messages and monitor them separately.

upvoted 0 times

...

Tequila

10 days ago

I think we should use an exception handling block in the Data Flow's DoFn code to push failed messages to a new Pub/Sub topic for monitoring.

upvoted 0 times

...