Dark Energy Survey

Dark Energy Survey#

Learning Objectives#

By the end of this project you should be able to:

describe the goals of the Dark Energy Survey (DES) and the data in its first public data release (DR1);
query a large astronomical database with SQL, applying data-quality selections and drawing a reproducible random sample;
recognize invalid (“sentinel”) values in a catalog and handle them properly;
separate stars from galaxies using a morphological classifier, and check the result with an independent quantity (colour); and
use K-means clustering to partition the survey footprint on the sky, and understand its limitations.

Overview#

The standard model of cosmology, the \(\Lambda\)CDM model, attributes about 70% of the total energy density of the Universe to dark energy. Its main effect is to make the expansion of the Universe accelerate, so measuring it accurately is vital to our understanding of the Universe.

The Dark Energy Survey (DES) [2], [3] is an international collaboration that carried out a large imaging survey of the southern sky. Its goals were to map hundreds of millions of galaxies, detect thousands of supernovae, and measure the patterns of cosmic structure, in order to determine the expansion history of the Universe and the nature of dark energy with high precision.

DES used the Dark Energy Camera (DECam) [4], a 570-megapixel camera with a 3 deg² field of view, mounted on the Victor M. Blanco 4-meter Telescope at the Cerro Tololo Inter-American Observatory (CTIO) in the Chilean Andes. DECam’s major components include a five-element optical corrector, a hexapod positioning and alignment system, a shutter, a set of colour filters, and the CCD imager.

In this project you will query the DES Data Release 1 (DR1) catalog [1], which contains nearly 400 million objects, directly from NOIRLab’s Astro Data Lab [6].

Data Sources#

DES DR1 [1] at NOIRLab’s Astro Data Lab: https://datalab.noirlab.edu/data/dark-energy-survey
Table browser for des_dr1.main, with descriptions of all 252 columns: https://datalab.noirlab.edu/data-explorer?showTable=des_dr1.main
Example notebook from Astro Data Lab on star/galaxy/QSO separation with DES DR1 [7], on which parts of this project are based.

Simple queries can be run anonymously, without a Data Lab account. If you use Data Lab or DES data in your work, follow the Data Lab acknowledgment guidelines and the disclaimers.

Importing and Setup#

You need the astro-datalab package to query Data Lab. On Google Colab, restart the runtime after installing it, then run the import cell.

%pip install --no-cache-dir astro-datalab

%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans

# Data Lab imports
from dl import queryClient as qc
from dl.helpers.utils import convert

Questions#

Answer each question in a markdown cell directly below it (add code cells where needed). For the background questions, aim for one to two well-organized paragraphs, and cite your sources. Your notebook should run from top to bottom without errors.

Part A: Background#

Question 01#

What is the Dark Energy Survey? What are its main goals? Which part of the sky does DES cover, and how large is that area? Why was that part of the sky chosen? Refs. [1], [2] and [3] are good starting points.

Question 02#

Read the DES DR1 paper [1], especially Section 1. What are the four complementary methods that DES uses to probe dark energy? Briefly explain what each measures. To achieve these goals, which surveys does DES conduct, over what area, and in which wavebands?

Question 03#

Which quantity does DES use to classify objects morphologically as stars or galaxies, as described in Section 4.5 of the DR1 paper [1]? Explain in your own words what SPREAD_MODEL measures [5], and what SPREADERR_MODEL is. Write down the definition of the classifier EXTENDED_COADD (their Eq. 2). What do its values 0, 1, 2 and 3 mean? For what magnitude range is its performance quoted in Table 1 of the paper?

Accessing Data from Data Lab#

Below is a sample query to retrieve a few columns from the des_dr1.main table, adapted from the Astro Data Lab example [7]. The query is written in SQL (Data Lab’s database is PostgreSQL).

# Write query statement (sql) as a string
# NOTE: triple quotes allow us to break the string on multiple lines
query = """
        SELECT mag_auto_g AS gmag, mag_auto_r AS rmag, mag_auto_i AS imag,
               mag_auto_z AS zmag, mag_auto_y AS ymag,
               kron_radius, spread_model_i, spreaderr_model_i, class_star_i,
               snr_g, snr_r, snr_i, snr_z, ra, dec
        FROM des_dr1.main
        WHERE (fluxerr_auto_g > 0 AND fluxerr_auto_r > 0 AND fluxerr_auto_z > 0)
        LIMIT 1000"""

# mag_auto_g,r,i,z,y     = AB magnitudes in DECam g,r,i,z,y bands (99 = no valid measurement)
# kron_radius            = Kron radius from SExtractor (pixels)
# spread_model_i         = star/galaxy classifier: light profile compared with the PSF, i band
# spreaderr_model_i      = uncertainty of spread_model_i
# class_star_i           = SExtractor star/extended-source classifier (0 = extended, 1 = point-like)
# snr_g,r,i,z            = signal-to-noise ratios (S/N) in g,r,i,z bands
# ra, dec                = celestial coordinates (degrees)
#
# WHERE: requirement that flux error > 0 (to avoid dividing by zero) in g,r,z bands
# LIMIT: returns the first 1,000 rows that satisfy the query

# Execute the query synchronously (fine for queries that take less than a few minutes).
# Convert the returned CSV-formatted string to a pandas DataFrame,
# making sure that any 'Infinity' values are converted to NaN.
sample_res = qc.query(sql=query)
sample_df = convert(sample_res, 'pandas', na_values=('Infinity', '-Infinity'))
sample_df.describe()

Part B: Querying and cleaning the data#

Question 04#

Write a query to retrieve the following columns from the des_dr1.main table (see the table browser for descriptions of these and all other columns):

coadd_object_id (a unique object identifier), ra, dec
mag_auto_X for X = g, r, i, z, y (AB magnitudes)
kron_radius
spread_model_i, spreaderr_model_i, class_star_i
snr_X for X = g, r, i, z

Apply the basic quality selection used in the DR1 paper (Sections 4.4–4.5):

flags_X < 4 and imaflags_iso_X = 0 for X = g, r, i, z (no serious extraction or image-masking problems);
mag_auto_i <= 22.5, the magnitude range for which the star/galaxy classifier was validated.

Random sampling. LIMIT N returns the first N rows that the database happens to find. That is fast, but not guaranteed to be a random or reproducible sample. Instead, sample the table with PostgreSQL’s TABLESAMPLE clause, placed directly after the table name:

FROM des_dr1.main TABLESAMPLE SYSTEM (0.07) REPEATABLE (42)

This selects about 0.07% of the table at random, with a fixed seed so that you get the same sample every time. The quality cuts reduce this to about 100,000 objects. The query takes up to about a minute.

Convert the result to a pandas DataFrame. Then:

Make a map of the objects on the sky (e.g. plt.hexbin of ra vs dec) and describe the DES footprint.
Plot the distributions of the magnitudes in each band. Some magnitudes take the value 99. What does that mean, and how should such values be handled? Are there other sentinel values in your data?

Question 05#

Stars and galaxies. The DES images are used to distinguish between stars (point sources), galaxies (resolved, if not too distant), and QSOs (which may have both point-like and resolved components).

kron_radius: Kron radius in pixels
spread_model_X: star–galaxy classifier comparing an extended model to the PSF model, for each band X
class_star_X: value ranging from 0.0 (not point-like) to 1.0 (point-like), for each band X

The value of spread_model is near zero for a point source (star or QSO), positive for an extended source (galaxy), and negative for an artifact smaller than the PSF (e.g. a bad pixel or cosmic ray).

Make a 2D histogram (e.g. plt.hexbin with bins='log') of spread_model_i versus mag_auto_i, zooming in to \(-0.02 <\) spread_model_i \(< 0.05\). Identify the stellar locus and the galaxies. What is the typical size of spread_model_i for galaxies? Why would a cut such as spread_model > 0.5 be meaningless?
Compute EXTENDED_COADD (Question 03) for every object. Count the objects in each class (0–3), and the numbers of stars (EXTENDED_COADD ≤ 1) and galaxies (EXTENDED_COADD ≥ 2) in your sample.
Check your classification with an independent property: colour. Make colour–colour plots (\(g-r\) versus \(r-i\)) separately for the stars and the galaxies. Stars follow a narrow “stellar locus”, while galaxies are more spread out. Does your classification look sensible? Also compare it with class_star_i.
How does the fraction of stars change with magnitude, and why does star/galaxy separation become harder for faint objects?

Part C: Partitioning the sky with K-means#

Question 06#

Suppose we wish to divide the DES footprint into regions, for example to process the data in parallel, or to define the sub-regions used to estimate statistical uncertainties with the jackknife method (DES uses K-means for this [8]). We will try to do this with K-means clustering on the sky positions of the objects.

In astronomy, right ascension (RA) and declination (Dec) are the coordinates on the sky that correspond to longitude and latitude on Earth. Make a DataFrame with the two columns ra and dec, and use K-means to fit the data with 2 clusters. Display the result with the helper function plot_clusters() below, and print the number of objects in each cluster.

Is the result what you expected? Is this a suitable way to divide the data into two equal parts?

def plot_clusters(data, labels):
    """Scatter plot of sky positions (first two columns of `data`), coloured by cluster label."""
    plt.figure(figsize=(10, 5))
    plt.scatter(data.iloc[:, 0], data.iloc[:, 1], s=1, c=labels, cmap='tab20')
    plt.xlabel('RA [degrees]')
    plt.ylabel('Dec [degrees]')
    plt.gca().set_aspect(1.)
    counts = np.bincount(labels)
    plt.title(f'{len(counts)} clusters; objects per cluster: min {counts.min()}, max {counts.max()}')

Question 07#

The DES footprint crosses RA = 0° = 360°. Modify the ra column so that if ra ≥ 180°, you subtract 360°, which makes the footprint continuous. Then:

Repeat the K-means fit with 2 clusters, and then with more clusters (e.g. 4, 8, 16, 32). For each value of n_clusters, quantify how equal the clusters are, e.g. the ratio of the smallest to the largest number of objects, or their coefficient of variation.
K-means minimizes Euclidean distances. Why is that not quite right for (RA, Dec) coordinates, especially far from the equator (hint: how long is one degree of RA at Dec = −60°)? Repeat the clustering using 3D unit vectors, \((x, y, z) = (\cos\delta\cos\alpha,\ \cos\delta\sin\alpha,\ \sin\delta)\), and compare.
Does K-means produce clusters with equal numbers of objects, equal areas, or neither? Explain why, in terms of what K-means optimizes.
For what n_clusters does the clustering make the most sense to you, and why? You shouldn’t increase n_clusters to an arbitrarily large number just to make the regions more uniform: each cluster represents a subset of data processed separately, and requiring more processors (or more jackknife regions) is not always feasible or sensible.

References#

[1] DES Collaboration (T. M. C. Abbott et al.), “The Dark Energy Survey: Data Release 1,” Astrophys. J. Suppl. Ser. 239, 18 (2018). doi:10.3847/1538-4365/aae9f0, arXiv:1801.03181

[2] The Dark Energy Survey Collaboration, “The Dark Energy Survey,” white paper (2005). arXiv:astro-ph/0510346

[3] DES Collaboration (T. Abbott et al.), “The Dark Energy Survey: more than dark energy – an overview,” Mon. Not. R. Astron. Soc. 460, 1270 (2016). doi:10.1093/mnras/stw641, arXiv:1601.00329

[4] B. Flaugher et al., “The Dark Energy Camera,” Astron. J. 150, 150 (2015). doi:10.1088/0004-6256/150/5/150, arXiv:1504.02900

[5] S. Desai et al., “The Blanco Cosmology Survey: Data Acquisition, Processing, Calibration, Quality Diagnostics, and Data Release,” Astrophys. J. 757, 83 (2012), which introduced SPREAD_MODEL. doi:10.1088/0004-637X/757/1/83, arXiv:1204.1210

[6] M. J. Fitzpatrick et al., “The NOAO Data Laboratory: a conceptual overview,” Proc. SPIE 9149, 91491T (2014). doi:10.1117/12.2057445

[7] Astro Data Lab example notebook, “Star/Galaxy/QSO separation with DES DR1.” astro-datalab/notebooks-latest

[8] E. Sheldon, kmeans_radec: K-means clustering on the sphere, used to define jackknife regions in DES analyses. esheldon/kmeans_radec

Acknowledgements#

Initial version: Ferzem Khan with some guidance from Mark Neubauer
Revised: 2026

This project uses data obtained from the Astro Data Lab at NSF NOIRLab, and public data from the Dark Energy Survey.