Tutorial: PSF Subtraction
Initial imports
[10]:
from datetime import datetime
from straklip import config, input_tables
from straklip.steps import psfsubtraction,klipphotometry
from straklip.stralog import getLogger
import os
import pkg_resources as pkg
First, we need to initialize the logger here.
[11]:
if 'SHARED_LOG_FILE' not in os.environ:
os.environ['SHARED_LOG_FILE'] = f'straklip_{datetime.now().strftime("%Y-%m-%d_%H%M")}.log'
getLogger('straklip', setup=True, logfile=os.environ['SHARED_LOG_FILE'],debu=False,
configfile=pkg.resource_filename('straklip', './config/logging.yaml'))
[11]:
<Logger straklip (DEBUG)>
Two pipeline configuration files are stored in the tutorials/pipeline_logs, directory, namely pipe.yaml and data.yaml. A more in-deep explanation of these file, is presented here: https://straklip.readthedocs.io/latest/quick_start.html. We start by loading into the pipeline as follow. Remember, to manipulate the options for each specific step, we can change the entries in the ‘pipe.yaml’ accordingly, or change them from a line of code like
pipe_cfg..analysis['steps']['extract_candidate']=True, before running the step. For this tutorial we will start form the previously generated dataframes from the pipeline.
[13]:
pipe_cfg='/Users/gstrampelli/StraKLIP/docs/source/tutorials/work/pipeline_logs/pipe.yaml' #or where these files are
data_cfg='/Users/gstrampelli/StraKLIP/docs/source/tutorials/work/pipeline_logs/data.yaml'
pipe_cfg = config.configure_pipeline(pipe_cfg,pipe_cfg=pipe_cfg,data_cfg=data_cfg,dt_string=datetime.now().strftime("%d/%m/%Y %H:%M:%S"))
data_cfg = config.configure_data(data_cfg,pipe_cfg)
2025-06-19 10:13:24 straklip.config :INFO (configure_pipeline:70[pid=56399])
==========================================================================================================================
StraKLIP pipeline started at date and time: 19/06/2025 10:13:24
Pipe_cfg: /Users/gstrampelli/StraKLIP/docs/source/tutorials/work/pipeline_logs/pipe.yaml
Data_cfg: /Users/gstrampelli/StraKLIP/docs/source/tutorials/work/pipeline_logs/data.yaml
==========================================================================================================================
2025-06-19 10:13:24 straklip.config :INFO (configure_data:151[pid=56399]) Validation of default labels and data successful!
Once the “pipe_cfg” and the “data_cfg” are configuration, we can load pre-existing dataframes.
In this case we use “skip_originals” True and “load” True to tell the pipeline tom not look for inpout catalogs, but for the dataframe already generated by the pipeline.
Loading the dataframe
[14]:
dataset = input_tables.Tables(data_cfg, pipe_cfg, skip_originals=True)
DF = config.configure_dataframe(dataset,load=True)
2025-06-19 10:13:29 straklip.config :WARNING (configure_dataframe:224[pid=56399]) get_Av_dict currently only supports VEGAmag system. Please provide your own set of AVs if in a differest system as AVs : {ext: {mag_filter : value}} in the data.yaml under target
2025-06-19 10:13:29 straklip.utils.ancillary :INFO (get_Av_dict:424[pid=56399]) before dust, V = 0.0 mag(VEGA)
2025-06-19 10:13:29 straklip.utils.ancillary :INFO (get_Av_dict:425[pid=56399]) after dust, V = 1.0146 mag(VEGA)
2025-06-19 10:13:29 straklip.utils.ancillary :INFO (get_Av_dict:444[pid=56399]) Av = 1.0146 mag
2025-06-19 10:13:29 straklip.utils.ancillary :INFO (get_Av_dict:488[pid=56399]) AV=0 wfc3,uvis2,f814w 0.0 mag(VEGA)
2025-06-19 10:13:29 straklip.utils.ancillary :INFO (get_Av_dict:489[pid=56399]) AV=1 wfc3,uvis2,f814w 0.6094 mag
2025-06-19 10:13:29 straklip.utils.ancillary :INFO (get_Av_dict:488[pid=56399]) AV=0 wfc3,uvis2,f850lp 0.0 mag(VEGA)
2025-06-19 10:13:29 straklip.utils.ancillary :INFO (get_Av_dict:489[pid=56399]) AV=1 wfc3,uvis2,f850lp 0.4694 mag
2025-06-19 10:13:29 straklip.config :INFO (configure_dataframe:270[pid=56399]) Fetching dataframes from /Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/out
[15]:
DF.keys
[15]:
['unq_targets',
'crossmatch_ids',
'mvs_targets',
'mvs_candidates',
'unq_candidates']
PSF subtraction
The next step wil perform PSF subtraction for each target in the “mvs_targets_df”, populating both the “mvs_candidates_df” and the “unq_candidates_df, creating a “residual_tiles” folder in the “out” directory with visual summary of the subtraction and updating the tile cube in the “mvs_tiles” and “median_tiles” with the outcome of the subtraction: a.k.a. the residuals for each KLIP mode and the models. This might take a while.
[16]:
psfsubtraction.run({'DF': DF, 'dataset': dataset})
2025-06-19 10:13:33 straklip.steps.psfsubtraction:INFO (run:229[pid=56399]) Making /Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/out/kliptemp to storage temporary PSF subtractions
2025-06-19 10:13:33 straklip.steps.psfsubtraction:INFO (run:236[pid=56399]) Performing KLIP PSF subtraction on tiles.
2025-06-19 10:13:33 straklip.config :INFO (make_paths:112[pid=56399]) Creating "/Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/out/residual_tiles/f814w"
2025-06-19 10:13:33 straklip.utils.ancillary :INFO (parallelization_package:968[pid=56399]) Max allowable workers 1, # of elements 1 , # of chunk 1 approx # of elemtent per chunks 1 (chunksize)
100%|██████████| 1/1 [00:00<00:00, 25.84it/s]
100%|██████████| 1/1 [00:00<00:00, 46.45it/s]
100%|██████████| 1/1 [00:00<00:00, 26.52it/s]
100%|██████████| 1/1 [00:00<00:00, 30.23it/s]
100%|██████████| 1/1 [00:00<00:00, 20.83it/s]
100%|██████████| 1/1 [00:00<00:00, 33.63it/s]
100%|██████████| 1/1 [00:00<00:00, 38.62it/s]
100%|██████████| 1/1 [00:00<00:00, 13.96it/s]
100%|██████████| 1/1 [00:00<00:00, 27.48it/s]
100%|██████████| 1/1 [00:00<00:00, 24.92it/s]
100%|██████████| 1/1 [00:00<00:00, 19.34it/s]
100%|██████████| 1/1 [00:00<00:00, 17.56it/s]
2025-06-19 10:14:58 straklip.config :INFO (make_paths:112[pid=56399]) Creating "/Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/out/residual_tiles/f850lp"
2025-06-19 10:14:58 straklip.utils.ancillary :INFO (parallelization_package:968[pid=56399]) Max allowable workers 1, # of elements 1 , # of chunk 1 approx # of elemtent per chunks 1 (chunksize)
100%|██████████| 1/1 [00:00<00:00, 54.15it/s]
100%|██████████| 1/1 [00:00<00:00, 9.09it/s]
100%|██████████| 1/1 [00:00<00:00, 43.64it/s]
100%|██████████| 1/1 [00:00<00:00, 20.05it/s]
100%|██████████| 1/1 [00:00<00:00, 27.63it/s]
100%|██████████| 1/1 [00:00<00:00, 20.64it/s]
100%|██████████| 1/1 [00:00<00:00, 17.90it/s]
100%|██████████| 1/1 [00:00<00:00, 63.23it/s]
100%|██████████| 1/1 [00:00<00:00, 25.69it/s]
100%|██████████| 1/1 [00:00<00:00, 42.05it/s]
100%|██████████| 1/1 [00:00<00:00, 9.90it/s]
100%|██████████| 1/1 [00:00<00:00, 42.54it/s]
2025-06-19 10:16:39 straklip.dataframe :INFO (save_dataframes:106[pid=56399]) Saving the the following keys in ['unq_targets_df', 'crossmatch_ids_df', 'mvs_targets_df', 'mvs_candidates_df', 'unq_candidates_df'] to .csv files in /Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/out
After generating the tiles for the residuals, we run the aperture photometry on each of them, to gather some basic information for the upcoming analysis. This step provide a basic aperture photometry and candidate identification (locating the brightest pixel in the residuals that persist across different filters and/or KLIP modes as requested).
Photmetry
With this step, a preliminary aperture photometry is performed on each companion. A more refined characterization of each companion can be performed on a later stage during the analysis. This might take a while
[17]:
klipphotometry.run({'DF': DF, 'dataset': dataset})
2025-06-19 10:16:46 straklip.steps.buildhdf :INFO (make_candidates_dataframes:22[pid=56399]) Creating the candidates dataframe
2025-06-19 10:16:46 straklip.steps.klipphotometry:INFO (update_candidate_dataframe:1181[pid=56399]) Updating the candidates dataframe
2025-06-19 10:16:46 straklip.steps.klipphotometry:INFO (update_companion_ZPT:882[pid=56399]) Working on the zero points for candidates
2025-06-19 10:16:46 straklip.utils.ancillary :INFO (parallelization_package:968[pid=56399]) Max allowable workers 8, # of elements 12 , # of chunk 12 approx # of elemtent per chunks 1 (chunksize)
2025-06-19 10:16:53 straklip.steps.klipphotometry:INFO (update_candidates:993[pid=56399]) Working on the candidates
2025-06-19 10:16:53 straklip.utils.ancillary :INFO (parallelization_package:968[pid=56399]) Max allowable workers 8, # of elements 12 , # of chunk 12 approx # of elemtent per chunks 1 (chunksize)
2025-06-19 10:17:03 straklip.steps.klipphotometry:INFO (update_candidate_dataframe:1200[pid=56399]) Updating the median tiles for the candidates
2025-06-19 10:17:03 straklip.steps.klipphotometry:INFO (update_median_candidates_tile:823[pid=56399]) Loading a total of 12 images
2025-06-19 10:17:03 straklip.steps.klipphotometry:INFO (update_median_candidates_tile:826[pid=56399]) Working in f814w:
2025-06-19 10:17:03 straklip.utils.ancillary :INFO (parallelization_package:968[pid=56399]) Max allowable workers 8, # of elements 12 , # of chunk 12 approx # of elemtent per chunks 1 (chunksize)
2025-06-19 10:17:19 straklip.steps.klipphotometry:INFO (update_median_candidates_tile:826[pid=56399]) Working in f850lp:
2025-06-19 10:17:19 straklip.utils.ancillary :INFO (parallelization_package:968[pid=56399]) Max allowable workers 8, # of elements 12 , # of chunk 12 approx # of elemtent per chunks 1 (chunksize)
2025-06-19 10:17:36 straklip.steps.klipphotometry:INFO (pruning_catalogs:1045[pid=56399]) Pruning the candidates df
2025-06-19 10:17:36 straklip.steps.klipphotometry:INFO (pruning_catalogs:1064[pid=56399]) Candidates df pruned.
2025-06-19 10:17:36 straklip.dataframe :INFO (save_dataframes:106[pid=56399]) Saving the the following keys in ['unq_targets_df', 'crossmatch_ids_df', 'mvs_targets_df', 'mvs_candidates_df', 'unq_candidates_df'] to .csv files in /Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/out
Now we can print the mvs_candidates_df and unq_candidates_df to look for highlighted candidates from the pipeline. A visual inspection of all the residuals is also recommended, since the automatic detection tool cna some time fail. Teh analysis can be run automatically on candidates in the _candidates_df, or manually providing the coordinates of the candidates and the id (see https://straklip.readthedocs.io/latest/tutorials/tutorial_analysis.html for examples).
[18]:
DF.mvs_candidates_df
[18]:
| mvs_ids | x_tile_f814w | x_tile_f850lp | y_tile_f814w | y_tile_f850lp | x_rot_f814w | x_rot_f850lp | y_rot_f814w | y_rot_f850lp | counts_f814w | ... | tp_above_th_f814w | tp_above_th_f850lp | tp_above_nsigma_f814w | tp_above_nsigma_f850lp | fp_above_th_f814w | fp_above_th_f850lp | fp_above_nsigma_f814w | fp_above_nsigma_f850lp | auc_f814w | auc_f850lp | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | NaN | 20.0 | NaN | 22.0 | NaN | 18.779 | NaN | 18.416 | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 1 | 8 | NaN | 18.0 | NaN | 19.0 | NaN | 22.212 | NaN | 19.676 | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 2 | 11 | 17.0 | 18.0 | 21.0 | 22.0 | 18.785 | 20.195 | 22.919 | 22.822 | 5104.492 | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
3 rows × 43 columns
[19]:
DF.unq_candidates_df
[19]:
| unq_ids | mass | emass | sep | mkmode | n_f814w | n_f850lp | nsigma_f814w | nsigma_f850lp | m_f814w | ... | tp_above_th_f814w | tp_above_th_f850lp | tp_above_nsigma_f814w | tp_above_nsigma_f850lp | fp_above_th_f814w | fp_above_th_f850lp | fp_above_nsigma_f814w | fp_above_nsigma_f850lp | auc_f814w | auc_f850lp | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | NaN | NaN | 2.000 | 9.0 | NaN | 1.0 | NaN | 20.433645 | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 1 | 8 | NaN | NaN | 2.236 | 8.0 | NaN | 1.0 | NaN | 9.857582 | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 2 | 11 | NaN | NaN | 2.995 | 8.0 | 1.0 | 1.0 | 109.63981 | 63.578743 | 22.477 | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
3 rows × 27 columns
Once the basic photometry has been performed on the candidates, we can move on on the final stage of the pipeline and perform a more in dept analysis and characterization of each system.