Unlock Your Databricks Potential: Master Databricks Certified Associate Developer for Apache Spark 3.5 - Python Now!

Correct : C

In Spark, a shuffle occurs when data needs to be redistributed across partitions or nodes, typically due to operations that require grouping, joining, or sorting.

groupBy triggers a shuffle because it aggregates data based on key values, requiring data from multiple partitions to be moved across the cluster.

Operations like filter and select are narrow transformations (no shuffle).

coalesce can reduce the number of partitions without a full shuffle (unlike repartition).

Why the other options are incorrect:

A (filter) -- Narrow transformation, no shuffle.

B (select) -- Simple projection, no data movement.

D (coalesce) -- Reduces partitions locally without shuffle.

Spark Architecture --- Transformations and Actions; narrow vs. wide transformations.

Databricks Exam Guide (June 2025): Section ''Apache Spark Architecture and Components'' --- explains shuffle operations and execution hierarchy.

===========

Correct : C

To define a micro-batch interval, the correct syntax is:

query = df.writeStream \

.outputMode('append') \

.trigger(processingTime='5 seconds') \

.start()

This schedules the query to execute every 5 seconds.

Continuous mode (used in Option A) is experimental and has limited sink support.

Option D is incorrect because processingTime must be a string (not an integer).

Option B triggers as fast as possible without interval control.

Correct : D

To ensure that data written out to disk is sorted, it is important to consider how Spark writes data when saving to Parquet tables. The methods .sort() or .orderBy() apply a global sort but do not guarantee that the sorting will persist in the final output files unless certain conditions are met (e.g. a single partition via .coalesce(1) --- which is not scalable).

Instead, the proper method in distributed Spark processing to ensure rows are sorted within their respective partitions when written out is:

.sortWithinPartitions('column_name')

According to Apache Spark documentation:

'sortWithinPartitions() ensures each partition is sorted by the specified columns. This is useful for downstream systems that require sorted files.'

This method works efficiently in distributed settings, avoids the performance bottleneck of global sorting (as in .orderBy() or .sort()), and guarantees each output partition has sorted records --- which meets the requirement of consistently sorted data.

Thus:

Option A and B do not guarantee the persisted file contents are sorted.

Option C introduces a bottleneck via .coalesce(1) (single partition).

Option D correctly applies sorting within partitions and is scalable.

Correct : C

If each of the 10 nodes runs 4 executors, and each executor is assigned 4 CPU cores:

Executors per node = 4

Cores per executor = 4

Total executors = 4 * 10 = 40

Total cores = 40 executors * 4 cores = 160 cores

However, Spark uses 1 core for overhead on each node when managing multiple executors. Thus, the practical allocation is:

Total usable executors = 4 executors/node 10 nodes = 40

Total cores = 4 cores 40 executors = 160

Answe r : A --- but the question asks specifically about ''CPU cores used by the application,'' assuming all

However, if you are considering 4 executors/node 4 cores = 16 cores per node, across 10 nodes, that's 160.

Final Answe r: A

Correct : D

In Apache Spark's execution hierarchy, the relationships are structured as follows:

Job: Created when an action (e.g., count(), collect(), save()) is triggered on an RDD or DataFrame.

Stage: Each job is divided into one or more stages, separated by shuffle boundaries (e.g., after a reduceByKey or join).

Task: Each stage consists of multiple tasks, one per partition, executed in parallel on executors.

Execution Hierarchy:

Job Stage(s) Task(s)

So, a job contains multiple stages, and each stage contains multiple tasks.

Why the other options are incorrect:

A: A job does not directly contain tasks without stages.

B: A stage cannot contain multiple jobs; it belongs to a single job.

C: Tasks do not contain jobs.

Reference (Databricks Apache Spark 3.5 -- Python / Study Guide):

Spark Architecture Overview --- Execution Hierarchy: Jobs, Stages, and Tasks.

Databricks Exam Guide (June 2025): Section ''Apache Spark Architecture and Components'' --- describes execution hierarchy and lazy evaluation.

===========