Google Exam Professional Data Engineer Topic 3 Question 97 Discussion

Actual exam question for Google's Professional Data Engineer exam

Question #: 97
Topic #: 3

[All Professional Data Engineer Questions]

A web server sends click events to a Pub/Sub topic as messages. The web server includes an event Timestamp attribute in the messages, which is the time when the click occurred. You have a Dataflow streaming job that reads from this Pub/Sub topic through a subscription, applies some transformations, and writes the result to another Pub/Sub topic for use by the advertising department. The advertising department needs to receive each message within 30 seconds of the corresponding click occurrence, but they report receiving the messages late. Your Dataflow job's system lag is about 5 seconds, and the data freshness is about 40 seconds. Inspecting a few messages show no more than 1 second lag between their event Timestamp and publish Time. What is the problem and what should you do?

AThe advertising department is causing delays when consuming the messages. Work with the advertising department to fix this.

BMessages in your Dataflow job are processed in less than 30 seconds, but your job cannot keep up with the backlog in the Pub/Sub
subscription. Optimize your job or increase the number of workers to fix this.

CThe web server is not pushing messages fast enough to Pub/Sub. Work with the web server team to fix this.

DMessages in your Dataflow job are taking more than 30 seconds to process. Optimize your job or increase the number of workers to fix this.

Show Suggested Answer

Suggested Answer: B

To ensure that the advertising department receives messages within 30 seconds of the click occurrence, and given the current system lag and data freshness metrics, the issue likely lies in the processing capacity of the Dataflow job. Here's why option B is the best choice:

System Lag and Data Freshness:

The system lag of 5 seconds indicates that Dataflow itself is processing messages relatively quickly.

However, the data freshness of 40 seconds suggests a significant delay before processing begins, indicating a backlog.

Backlog in Pub/Sub Subscription:

A backlog occurs when the rate of incoming messages exceeds the rate at which the Dataflow job can process them, causing delays.

Optimizing the Dataflow Job:

To handle the incoming message rate, the Dataflow job needs to be optimized or scaled up by increasing the number of workers, ensuring it can keep up with the message inflow.

Steps to Implement:

Analyze the Dataflow Job:

Inspect the Dataflow job metrics to identify bottlenecks and inefficiencies.

Optimize Processing Logic:

Optimize the transformations and operations within the Dataflow pipeline to improve processing efficiency.

Increase Number of Workers:

Scale the Dataflow job by increasing the number of workers to handle the higher load, reducing the backlog.

Dataflow Monitoring

Scaling Dataflow Jobs

by Kristin at Sep 16, 2024, 08:56 AM

Limited Time Offer

25%

Off

Get Premium Professional Data Engineer Questions as Interactive Web-Based Practice Test or PDF

Contribute your Thoughts:

Submit Cancel

Jaleesa

29 days ago

The web server team's gonna be like, 'It's not us, it's you!' But Option D is the way to go. Time to get that Dataflow job running like a well-oiled machine.

upvoted 0 times

...

Mattie

1 months ago

Gotta love these Pub/Sub questions. They always seem to have a twist, don't they? I'm going with Option D. Optimize that Dataflow job, baby!

upvoted 0 times

...

Bettina

1 months ago

Option B seems like the way to go. The Dataflow job can't keep up with the backlog in the Pub/Sub subscription. Time to scale up those workers!

upvoted 0 times

Lea

2 days ago

B) Messages in your Dataflow job are processed in less than 30 seconds, but your job cannot keep up with the backlog in the Pub/Sub subscription. Optimize your job or increase the number of workers to fix this.

upvoted 0 times

...

Nickole

1 months ago

Hmm, I'm not sure the advertising department is the issue here. It sounds like the Dataflow job is the bottleneck. I'd go with Option D and optimize the job.

upvoted 0 times

...

Nikita

1 months ago

But what if the advertising department is causing delays when consuming the messages? Shouldn't we work with them to fix this?

upvoted 0 times

...

Lino

1 months ago

The problem is clearly with the Dataflow job. It's taking too long to process the messages, which is causing the delay for the advertising department. Option D is the correct answer.

upvoted 0 times