Tutorial: Joining DRC catalogues
The goal of this tutorial is to show how to build a mvs_targets_catalogue and a unq_targets_catalogue in a format that the pipeline can digest. In this tutorial we assume we are working with DRC images, so the mvs_input_targets_df and unq_input_targets_df will be the same. If working with FLC images instead, other route can be explored to achieve the similar results. Please reach out to: gstrampelli@stsci.edu for a sample data to rune this tutorial
Setup and imports
[1]:
import os
import pandas as pd
from glob import glob
from tqdm import tqdm
from astropy.io import fits
import datetime
import time
import numpy as np
from straklip.utils.utils_tile import small_tiles
Loading input catalog
Before running the pipeline, we need a series of HST _flc or _flt_ images, and a catalog recording the x, y coordinates and a few additional information of each sources on these images.
The mandatory columns for the mvs_dataframe catalog are the following:
- unq_ids: ids for average catalog
- mvs_ids: ids for multivisit catalog
- vis: visit column name in catalog
- ext: extension column name in catalog that identify SCI in fits file (for HST if CCDCHIP = 1, EXT = 4, CCDCHIP = 2, EXT = 1)
- x: filter wise x column name in catalog
- y: filter wise y column name in catalog
- fitsroot: filter wise fitsroot column name in catalog (it's the filename without the `_flc` extension)
- exptime: filter wise exposure time for each source, i.e. header['EXPTIME']
- pav3: filter wise the HST V3 position angle, i.e. header['PA_V3']
- rota: filter wise HST orientation, i.e. header['ORIENTAT']
The section mvs_table in the data.yaml can be used to tell the pipeline how to match your columns name with the pipeline default (stored in the pipe.yaml).
An unique catalog recording the ra, dec and type of each unique source is also need. The mandatory columns for the unq_dataframe catalog are the following:
- unq_ids: ids for average catalog
- ra: ra column name in catalog
- dec: dec column name in catalog
- type: filter wise type column name in catalog (see below)
Having the photometry can help, but it’s not mandatory. If needed, the pipeline can perform its own aperture photometry. To generate these two tables, we first load the default catalogue targets_drc.csv and we assign to each target with the same name in the targets_drc target columns, the same unq_ids
[2]:
unq_input_target_df=pd.read_csv('/Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/data/targets_drc.csv')
unq_input_target_df
[2]:
| target | file | filter | ra | dec | x | y | mag_aper | e_mag_aper | dist | snr | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | J155150.21-213457.4 | iexn13010_drc.fits | F814W | 237.959171 | -21.582686 | 766.682062 | 870.519962 | 22.277353 | 0.021258 | 1.416799 | 227.932175 |
| 1 | J164636.12-231337.6 | iexn01010_drc.fits | F814W | 251.650498 | -23.227155 | 769.100793 | 866.146099 | 20.840983 | 0.005901 | 4.008029 | 792.416809 |
| 2 | J164636.12-231337.6 | iexn01020_drc.fits | F850LP | 251.650498 | -23.227156 | 768.870891 | 866.488792 | 19.749264 | 0.006932 | 3.617601 | 600.396423 |
| 3 | J155416.68-263018.1 | iexn02010_drc.fits | F814W | 238.569423 | -26.505131 | 762.415167 | 869.798303 | 21.448087 | 0.010977 | 5.588474 | 411.439362 |
| 4 | J155416.68-263018.1 | iexn02020_drc.fits | F850LP | 238.569423 | -26.505132 | 762.213142 | 870.180384 | 20.330281 | 0.012019 | 5.789669 | 338.973694 |
| 5 | J160644.67-203342.8 | iexn03010_drc.fits | F814W | 241.686092 | -20.561958 | 765.630228 | 867.979551 | 21.493397 | 0.011936 | 3.114166 | 387.477783 |
| 6 | J160644.67-203342.8 | iexn03020_drc.fits | F850LP | 241.686094 | -20.561958 | 765.408206 | 868.467341 | 20.318072 | 0.012070 | 3.011052 | 357.040802 |
| 7 | J161434.76-241933.4 | iexn04010_drc.fits | F814W | 243.644804 | -24.326008 | 767.251059 | 867.473929 | 21.553108 | 0.011095 | 2.634757 | 406.735779 |
| 8 | J161434.76-241933.4 | iexn04020_drc.fits | F850LP | 243.644805 | -24.326008 | 767.009600 | 867.939866 | 20.299904 | 0.011329 | 2.285835 | 361.328217 |
| 9 | J162810.30-264024.2 | iexn05010_drc.fits | F814W | 247.042901 | -26.673437 | 766.601507 | 868.267308 | 21.419805 | 0.010367 | 2.226657 | 451.694641 |
| 10 | J162810.30-264024.2 | iexn05020_drc.fits | F850LP | 247.042902 | -26.673437 | 766.336710 | 868.601284 | 20.159706 | 0.010190 | 2.173232 | 401.216766 |
| 11 | J160731.61-214654.6 | iexn06010_drc.fits | F814W | 241.881676 | -21.781889 | 767.081804 | 866.901849 | 22.182586 | 0.022100 | 3.231350 | 215.942444 |
| 12 | J160731.61-214654.6 | iexn06020_drc.fits | F850LP | 241.881676 | -21.781889 | 766.867054 | 867.166727 | 20.929665 | 0.021179 | 3.051393 | 187.888596 |
| 13 | J155543.75-232028.9 | iexn07010_drc.fits | F814W | 238.932269 | -23.341438 | 765.825833 | 868.625604 | 21.854202 | 0.015996 | 2.572152 | 284.837280 |
| 14 | J155543.75-232028.9 | iexn07020_drc.fits | F850LP | 238.932269 | -23.341438 | 765.608929 | 869.098238 | 20.710080 | 0.016869 | 2.555464 | 252.469177 |
| 15 | J154915.32-244139.1 | iexn08010_drc.fits | F814W | 237.313809 | -24.694252 | 765.740929 | 868.295724 | 22.007757 | 0.017982 | 2.829834 | 249.846924 |
| 16 | J154915.32-244139.1 | iexn08020_drc.fits | F850LP | 237.313810 | -24.694252 | 765.619928 | 868.761725 | 20.915487 | 0.020411 | 2.682921 | 199.514282 |
| 17 | J160918.67-222923.9 | iexn09010_drc.fits | F814W | 242.327752 | -22.490052 | 762.743761 | 866.727766 | 22.001171 | 0.018724 | 6.191572 | 242.439774 |
| 18 | J160918.67-222923.9 | iexn09020_drc.fits | F850LP | 242.327752 | -22.490052 | 762.620949 | 866.796443 | 20.883590 | 0.020491 | 6.260748 | 191.753143 |
| 19 | J160656.33-165644.6 | iexn10010_drc.fits | F814W | 241.734689 | -16.945773 | 767.150769 | 866.806861 | 22.110921 | 0.020702 | 3.304138 | 217.606400 |
| 20 | J160656.33-165644.6 | iexn10020_drc.fits | F850LP | 241.734691 | -16.945775 | 766.981161 | 867.158018 | 20.930442 | 0.021017 | 3.019088 | 201.309143 |
| 21 | J155150.21-213457.4 | iexn13020_drc.fits | F850LP | 237.959171 | -21.582688 | 766.297865 | 870.863230 | 21.078353 | 0.022984 | 1.908515 | 172.553406 |
| 22 | J042705.86+261520.3 | iexn54010_drc.fits | F814W | 66.774431 | 26.255547 | 775.939388 | 859.286880 | 21.223795 | 0.008219 | 13.334347 | 413.132355 |
| 23 | J042705.86+261520.3 | iexn54020_drc.fits | F850LP | 66.774431 | 26.255554 | 775.359033 | 859.690031 | 20.284122 | 0.010910 | 12.666919 | 302.929840 |
[3]:
mvs_input_target_df=pd.read_csv('/Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/data/targets_drc.csv')
mvs_input_target_df
[3]:
| target | file | filter | ra | dec | x | y | mag_aper | e_mag_aper | dist | snr | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | J155150.21-213457.4 | iexn13010_drc.fits | F814W | 237.959171 | -21.582686 | 766.682062 | 870.519962 | 22.277353 | 0.021258 | 1.416799 | 227.932175 |
| 1 | J164636.12-231337.6 | iexn01010_drc.fits | F814W | 251.650498 | -23.227155 | 769.100793 | 866.146099 | 20.840983 | 0.005901 | 4.008029 | 792.416809 |
| 2 | J164636.12-231337.6 | iexn01020_drc.fits | F850LP | 251.650498 | -23.227156 | 768.870891 | 866.488792 | 19.749264 | 0.006932 | 3.617601 | 600.396423 |
| 3 | J155416.68-263018.1 | iexn02010_drc.fits | F814W | 238.569423 | -26.505131 | 762.415167 | 869.798303 | 21.448087 | 0.010977 | 5.588474 | 411.439362 |
| 4 | J155416.68-263018.1 | iexn02020_drc.fits | F850LP | 238.569423 | -26.505132 | 762.213142 | 870.180384 | 20.330281 | 0.012019 | 5.789669 | 338.973694 |
| 5 | J160644.67-203342.8 | iexn03010_drc.fits | F814W | 241.686092 | -20.561958 | 765.630228 | 867.979551 | 21.493397 | 0.011936 | 3.114166 | 387.477783 |
| 6 | J160644.67-203342.8 | iexn03020_drc.fits | F850LP | 241.686094 | -20.561958 | 765.408206 | 868.467341 | 20.318072 | 0.012070 | 3.011052 | 357.040802 |
| 7 | J161434.76-241933.4 | iexn04010_drc.fits | F814W | 243.644804 | -24.326008 | 767.251059 | 867.473929 | 21.553108 | 0.011095 | 2.634757 | 406.735779 |
| 8 | J161434.76-241933.4 | iexn04020_drc.fits | F850LP | 243.644805 | -24.326008 | 767.009600 | 867.939866 | 20.299904 | 0.011329 | 2.285835 | 361.328217 |
| 9 | J162810.30-264024.2 | iexn05010_drc.fits | F814W | 247.042901 | -26.673437 | 766.601507 | 868.267308 | 21.419805 | 0.010367 | 2.226657 | 451.694641 |
| 10 | J162810.30-264024.2 | iexn05020_drc.fits | F850LP | 247.042902 | -26.673437 | 766.336710 | 868.601284 | 20.159706 | 0.010190 | 2.173232 | 401.216766 |
| 11 | J160731.61-214654.6 | iexn06010_drc.fits | F814W | 241.881676 | -21.781889 | 767.081804 | 866.901849 | 22.182586 | 0.022100 | 3.231350 | 215.942444 |
| 12 | J160731.61-214654.6 | iexn06020_drc.fits | F850LP | 241.881676 | -21.781889 | 766.867054 | 867.166727 | 20.929665 | 0.021179 | 3.051393 | 187.888596 |
| 13 | J155543.75-232028.9 | iexn07010_drc.fits | F814W | 238.932269 | -23.341438 | 765.825833 | 868.625604 | 21.854202 | 0.015996 | 2.572152 | 284.837280 |
| 14 | J155543.75-232028.9 | iexn07020_drc.fits | F850LP | 238.932269 | -23.341438 | 765.608929 | 869.098238 | 20.710080 | 0.016869 | 2.555464 | 252.469177 |
| 15 | J154915.32-244139.1 | iexn08010_drc.fits | F814W | 237.313809 | -24.694252 | 765.740929 | 868.295724 | 22.007757 | 0.017982 | 2.829834 | 249.846924 |
| 16 | J154915.32-244139.1 | iexn08020_drc.fits | F850LP | 237.313810 | -24.694252 | 765.619928 | 868.761725 | 20.915487 | 0.020411 | 2.682921 | 199.514282 |
| 17 | J160918.67-222923.9 | iexn09010_drc.fits | F814W | 242.327752 | -22.490052 | 762.743761 | 866.727766 | 22.001171 | 0.018724 | 6.191572 | 242.439774 |
| 18 | J160918.67-222923.9 | iexn09020_drc.fits | F850LP | 242.327752 | -22.490052 | 762.620949 | 866.796443 | 20.883590 | 0.020491 | 6.260748 | 191.753143 |
| 19 | J160656.33-165644.6 | iexn10010_drc.fits | F814W | 241.734689 | -16.945773 | 767.150769 | 866.806861 | 22.110921 | 0.020702 | 3.304138 | 217.606400 |
| 20 | J160656.33-165644.6 | iexn10020_drc.fits | F850LP | 241.734691 | -16.945775 | 766.981161 | 867.158018 | 20.930442 | 0.021017 | 3.019088 | 201.309143 |
| 21 | J155150.21-213457.4 | iexn13020_drc.fits | F850LP | 237.959171 | -21.582688 | 766.297865 | 870.863230 | 21.078353 | 0.022984 | 1.908515 | 172.553406 |
| 22 | J042705.86+261520.3 | iexn54010_drc.fits | F814W | 66.774431 | 26.255547 | 775.939388 | 859.286880 | 21.223795 | 0.008219 | 13.334347 | 413.132355 |
| 23 | J042705.86+261520.3 | iexn54020_drc.fits | F850LP | 66.774431 | 26.255554 | 775.359033 | 859.690031 | 20.284122 | 0.010910 | 12.666919 | 302.929840 |
[4]:
elno=0
# mvs_input_target_df['unq_ids'] = np.nan
for target in mvs_input_target_df.target.unique():
mvs_input_target_df.loc[mvs_input_target_df.target == target, 'unq_ids'] = int(elno)
elno+=1
mvs_input_target_df=mvs_input_target_df.sort_values('unq_ids').reset_index(drop=True)
mvs_input_target_df
[4]:
| target | file | filter | ra | dec | x | y | mag_aper | e_mag_aper | dist | snr | unq_ids | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | J155150.21-213457.4 | iexn13010_drc.fits | F814W | 237.959171 | -21.582686 | 766.682062 | 870.519962 | 22.277353 | 0.021258 | 1.416799 | 227.932175 | 0.0 |
| 1 | J155150.21-213457.4 | iexn13020_drc.fits | F850LP | 237.959171 | -21.582688 | 766.297865 | 870.863230 | 21.078353 | 0.022984 | 1.908515 | 172.553406 | 0.0 |
| 2 | J164636.12-231337.6 | iexn01010_drc.fits | F814W | 251.650498 | -23.227155 | 769.100793 | 866.146099 | 20.840983 | 0.005901 | 4.008029 | 792.416809 | 1.0 |
| 3 | J164636.12-231337.6 | iexn01020_drc.fits | F850LP | 251.650498 | -23.227156 | 768.870891 | 866.488792 | 19.749264 | 0.006932 | 3.617601 | 600.396423 | 1.0 |
| 4 | J155416.68-263018.1 | iexn02010_drc.fits | F814W | 238.569423 | -26.505131 | 762.415167 | 869.798303 | 21.448087 | 0.010977 | 5.588474 | 411.439362 | 2.0 |
| 5 | J155416.68-263018.1 | iexn02020_drc.fits | F850LP | 238.569423 | -26.505132 | 762.213142 | 870.180384 | 20.330281 | 0.012019 | 5.789669 | 338.973694 | 2.0 |
| 6 | J160644.67-203342.8 | iexn03010_drc.fits | F814W | 241.686092 | -20.561958 | 765.630228 | 867.979551 | 21.493397 | 0.011936 | 3.114166 | 387.477783 | 3.0 |
| 7 | J160644.67-203342.8 | iexn03020_drc.fits | F850LP | 241.686094 | -20.561958 | 765.408206 | 868.467341 | 20.318072 | 0.012070 | 3.011052 | 357.040802 | 3.0 |
| 8 | J161434.76-241933.4 | iexn04010_drc.fits | F814W | 243.644804 | -24.326008 | 767.251059 | 867.473929 | 21.553108 | 0.011095 | 2.634757 | 406.735779 | 4.0 |
| 9 | J161434.76-241933.4 | iexn04020_drc.fits | F850LP | 243.644805 | -24.326008 | 767.009600 | 867.939866 | 20.299904 | 0.011329 | 2.285835 | 361.328217 | 4.0 |
| 10 | J162810.30-264024.2 | iexn05010_drc.fits | F814W | 247.042901 | -26.673437 | 766.601507 | 868.267308 | 21.419805 | 0.010367 | 2.226657 | 451.694641 | 5.0 |
| 11 | J162810.30-264024.2 | iexn05020_drc.fits | F850LP | 247.042902 | -26.673437 | 766.336710 | 868.601284 | 20.159706 | 0.010190 | 2.173232 | 401.216766 | 5.0 |
| 12 | J160731.61-214654.6 | iexn06010_drc.fits | F814W | 241.881676 | -21.781889 | 767.081804 | 866.901849 | 22.182586 | 0.022100 | 3.231350 | 215.942444 | 6.0 |
| 13 | J160731.61-214654.6 | iexn06020_drc.fits | F850LP | 241.881676 | -21.781889 | 766.867054 | 867.166727 | 20.929665 | 0.021179 | 3.051393 | 187.888596 | 6.0 |
| 14 | J155543.75-232028.9 | iexn07010_drc.fits | F814W | 238.932269 | -23.341438 | 765.825833 | 868.625604 | 21.854202 | 0.015996 | 2.572152 | 284.837280 | 7.0 |
| 15 | J155543.75-232028.9 | iexn07020_drc.fits | F850LP | 238.932269 | -23.341438 | 765.608929 | 869.098238 | 20.710080 | 0.016869 | 2.555464 | 252.469177 | 7.0 |
| 16 | J154915.32-244139.1 | iexn08010_drc.fits | F814W | 237.313809 | -24.694252 | 765.740929 | 868.295724 | 22.007757 | 0.017982 | 2.829834 | 249.846924 | 8.0 |
| 17 | J154915.32-244139.1 | iexn08020_drc.fits | F850LP | 237.313810 | -24.694252 | 765.619928 | 868.761725 | 20.915487 | 0.020411 | 2.682921 | 199.514282 | 8.0 |
| 18 | J160918.67-222923.9 | iexn09010_drc.fits | F814W | 242.327752 | -22.490052 | 762.743761 | 866.727766 | 22.001171 | 0.018724 | 6.191572 | 242.439774 | 9.0 |
| 19 | J160918.67-222923.9 | iexn09020_drc.fits | F850LP | 242.327752 | -22.490052 | 762.620949 | 866.796443 | 20.883590 | 0.020491 | 6.260748 | 191.753143 | 9.0 |
| 20 | J160656.33-165644.6 | iexn10010_drc.fits | F814W | 241.734689 | -16.945773 | 767.150769 | 866.806861 | 22.110921 | 0.020702 | 3.304138 | 217.606400 | 10.0 |
| 21 | J160656.33-165644.6 | iexn10020_drc.fits | F850LP | 241.734691 | -16.945775 | 766.981161 | 867.158018 | 20.930442 | 0.021017 | 3.019088 | 201.309143 | 10.0 |
| 22 | J042705.86+261520.3 | iexn54010_drc.fits | F814W | 66.774431 | 26.255547 | 775.939388 | 859.286880 | 21.223795 | 0.008219 | 13.334347 | 413.132355 | 11.0 |
| 23 | J042705.86+261520.3 | iexn54020_drc.fits | F850LP | 66.774431 | 26.255554 | 775.359033 | 859.690031 | 20.284122 | 0.010910 | 12.666919 | 302.929840 | 11.0 |
[5]:
for target in mvs_input_target_df.target.unique():
if unq_input_target_df.loc[unq_input_target_df.target ==target].empty:
index=mvs_input_target_df.loc[(mvs_input_target_df.target==target)].index
mvs_input_target_df.drop(index,inplace=True)
else:
unq_input_target_df.loc[unq_input_target_df.target ==target,'unq_ids'] = mvs_input_target_df.loc[mvs_input_target_df.target == target, 'unq_ids'].values[0]
unq_input_target_df=unq_input_target_df.sort_values('unq_ids').reset_index(drop=True)
unq_input_target_df
[5]:
| target | file | filter | ra | dec | x | y | mag_aper | e_mag_aper | dist | snr | unq_ids | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | J155150.21-213457.4 | iexn13010_drc.fits | F814W | 237.959171 | -21.582686 | 766.682062 | 870.519962 | 22.277353 | 0.021258 | 1.416799 | 227.932175 | 0.0 |
| 1 | J155150.21-213457.4 | iexn13020_drc.fits | F850LP | 237.959171 | -21.582688 | 766.297865 | 870.863230 | 21.078353 | 0.022984 | 1.908515 | 172.553406 | 0.0 |
| 2 | J164636.12-231337.6 | iexn01010_drc.fits | F814W | 251.650498 | -23.227155 | 769.100793 | 866.146099 | 20.840983 | 0.005901 | 4.008029 | 792.416809 | 1.0 |
| 3 | J164636.12-231337.6 | iexn01020_drc.fits | F850LP | 251.650498 | -23.227156 | 768.870891 | 866.488792 | 19.749264 | 0.006932 | 3.617601 | 600.396423 | 1.0 |
| 4 | J155416.68-263018.1 | iexn02010_drc.fits | F814W | 238.569423 | -26.505131 | 762.415167 | 869.798303 | 21.448087 | 0.010977 | 5.588474 | 411.439362 | 2.0 |
| 5 | J155416.68-263018.1 | iexn02020_drc.fits | F850LP | 238.569423 | -26.505132 | 762.213142 | 870.180384 | 20.330281 | 0.012019 | 5.789669 | 338.973694 | 2.0 |
| 6 | J160644.67-203342.8 | iexn03010_drc.fits | F814W | 241.686092 | -20.561958 | 765.630228 | 867.979551 | 21.493397 | 0.011936 | 3.114166 | 387.477783 | 3.0 |
| 7 | J160644.67-203342.8 | iexn03020_drc.fits | F850LP | 241.686094 | -20.561958 | 765.408206 | 868.467341 | 20.318072 | 0.012070 | 3.011052 | 357.040802 | 3.0 |
| 8 | J161434.76-241933.4 | iexn04010_drc.fits | F814W | 243.644804 | -24.326008 | 767.251059 | 867.473929 | 21.553108 | 0.011095 | 2.634757 | 406.735779 | 4.0 |
| 9 | J161434.76-241933.4 | iexn04020_drc.fits | F850LP | 243.644805 | -24.326008 | 767.009600 | 867.939866 | 20.299904 | 0.011329 | 2.285835 | 361.328217 | 4.0 |
| 10 | J162810.30-264024.2 | iexn05010_drc.fits | F814W | 247.042901 | -26.673437 | 766.601507 | 868.267308 | 21.419805 | 0.010367 | 2.226657 | 451.694641 | 5.0 |
| 11 | J162810.30-264024.2 | iexn05020_drc.fits | F850LP | 247.042902 | -26.673437 | 766.336710 | 868.601284 | 20.159706 | 0.010190 | 2.173232 | 401.216766 | 5.0 |
| 12 | J160731.61-214654.6 | iexn06010_drc.fits | F814W | 241.881676 | -21.781889 | 767.081804 | 866.901849 | 22.182586 | 0.022100 | 3.231350 | 215.942444 | 6.0 |
| 13 | J160731.61-214654.6 | iexn06020_drc.fits | F850LP | 241.881676 | -21.781889 | 766.867054 | 867.166727 | 20.929665 | 0.021179 | 3.051393 | 187.888596 | 6.0 |
| 14 | J155543.75-232028.9 | iexn07010_drc.fits | F814W | 238.932269 | -23.341438 | 765.825833 | 868.625604 | 21.854202 | 0.015996 | 2.572152 | 284.837280 | 7.0 |
| 15 | J155543.75-232028.9 | iexn07020_drc.fits | F850LP | 238.932269 | -23.341438 | 765.608929 | 869.098238 | 20.710080 | 0.016869 | 2.555464 | 252.469177 | 7.0 |
| 16 | J154915.32-244139.1 | iexn08010_drc.fits | F814W | 237.313809 | -24.694252 | 765.740929 | 868.295724 | 22.007757 | 0.017982 | 2.829834 | 249.846924 | 8.0 |
| 17 | J154915.32-244139.1 | iexn08020_drc.fits | F850LP | 237.313810 | -24.694252 | 765.619928 | 868.761725 | 20.915487 | 0.020411 | 2.682921 | 199.514282 | 8.0 |
| 18 | J160918.67-222923.9 | iexn09010_drc.fits | F814W | 242.327752 | -22.490052 | 762.743761 | 866.727766 | 22.001171 | 0.018724 | 6.191572 | 242.439774 | 9.0 |
| 19 | J160918.67-222923.9 | iexn09020_drc.fits | F850LP | 242.327752 | -22.490052 | 762.620949 | 866.796443 | 20.883590 | 0.020491 | 6.260748 | 191.753143 | 9.0 |
| 20 | J160656.33-165644.6 | iexn10010_drc.fits | F814W | 241.734689 | -16.945773 | 767.150769 | 866.806861 | 22.110921 | 0.020702 | 3.304138 | 217.606400 | 10.0 |
| 21 | J160656.33-165644.6 | iexn10020_drc.fits | F850LP | 241.734691 | -16.945775 | 766.981161 | 867.158018 | 20.930442 | 0.021017 | 3.019088 | 201.309143 | 10.0 |
| 22 | J042705.86+261520.3 | iexn54010_drc.fits | F814W | 66.774431 | 26.255547 | 775.939388 | 859.286880 | 21.223795 | 0.008219 | 13.334347 | 413.132355 | 11.0 |
| 23 | J042705.86+261520.3 | iexn54020_drc.fits | F850LP | 66.774431 | 26.255554 | 775.359033 | 859.690031 | 20.284122 | 0.010910 | 12.666919 | 302.929840 | 11.0 |
Lets populate the mvs_targets_df
The StraKLIP pipeline requires a series on variables in the mvs_targets_df that can be loaded from the fits header of each image
[6]:
path2fits='/Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/data/fits/'
for file in tqdm(glob(path2fits+'/*')):
fitsname=file.split('/')[-1].split('.')[0]
hdul=fits.open(path2fits+fitsname+'.fits')
mvs_input_target_df.loc[mvs_input_target_df.file==fitsname+'.fits','filters']=hdul[0].header['FILTER']
dateobs=hdul[0].header['DATE-OBS'].split('-')
timeobs=hdul[0].header['TIME-OBS'].split(':')
# df=pd.read_csv(file)
date_time = datetime.datetime(int(dateobs[0]), int(dateobs[1]), int(dateobs[2]), int(timeobs[0]), int(timeobs[1]), int(timeobs[2]))
mvs_input_target_df.loc[mvs_input_target_df.file==fitsname+'.fits','unittime']=time.mktime(date_time.timetuple())
mvs_input_target_df.loc[mvs_input_target_df.file==fitsname+'.fits','fitsroot']=fitsname.split('_')[0]
mvs_input_target_df.loc[mvs_input_target_df.file==fitsname+'.fits','filters']=hdul[0].header['FILTER']
mvs_input_target_df.loc[mvs_input_target_df.file==fitsname+'.fits','ccd']=2
mvs_input_target_df.loc[mvs_input_target_df.file==fitsname+'.fits','visit']=str(fitsname[4:6])
mvs_input_target_df.loc[mvs_input_target_df.file==fitsname+'.fits','exptime']=hdul[0].header['EXPTIME']
mvs_input_target_df.loc[mvs_input_target_df.file==fitsname+'.fits','pav3']=hdul[0].header['PA_V3']
mvs_input_target_df.loc[mvs_input_target_df.file==fitsname+'.fits','rota']=hdul[1].header['ORIENTAT']
mvs_input_target_df
100%|██████████| 24/24 [00:00<00:00, 366.36it/s]
[6]:
| target | file | filter | ra | dec | x | y | mag_aper | e_mag_aper | dist | snr | unq_ids | filters | unittime | fitsroot | ccd | visit | exptime | pav3 | rota | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | J155150.21-213457.4 | iexn13010_drc.fits | F814W | 237.959171 | -21.582686 | 766.682062 | 870.519962 | 22.277353 | 0.021258 | 1.416799 | 227.932175 | 0.0 | F814W | 1.679273e+09 | iexn13010 | 2.0 | 13 | 712.0 | 124.436996 | 169.438303 |
| 1 | J155150.21-213457.4 | iexn13020_drc.fits | F850LP | 237.959171 | -21.582688 | 766.297865 | 870.863230 | 21.078353 | 0.022984 | 1.908515 | 172.553406 | 0.0 | F850LP | 1.679275e+09 | iexn13020 | 2.0 | 13 | 712.0 | 124.436996 | 169.438303 |
| 2 | J164636.12-231337.6 | iexn01010_drc.fits | F814W | 251.650498 | -23.227155 | 769.100793 | 866.146099 | 20.840983 | 0.005901 | 4.008029 | 792.416809 | 1.0 | F814W | 1.678937e+09 | iexn01010 | 2.0 | 01 | 712.0 | 97.360497 | 142.364009 |
| 3 | J164636.12-231337.6 | iexn01020_drc.fits | F850LP | 251.650498 | -23.227156 | 768.870891 | 866.488792 | 19.749264 | 0.006932 | 3.617601 | 600.396423 | 1.0 | F850LP | 1.678938e+09 | iexn01020 | 2.0 | 01 | 712.0 | 97.360497 | 142.364009 |
| 4 | J155416.68-263018.1 | iexn02010_drc.fits | F814W | 238.569423 | -26.505131 | 762.415167 | 869.798303 | 21.448087 | 0.010977 | 5.588474 | 411.439362 | 2.0 | F814W | 1.674509e+09 | iexn02010 | 2.0 | 02 | 716.0 | 98.681168 | 143.685147 |
| 5 | J155416.68-263018.1 | iexn02020_drc.fits | F850LP | 238.569423 | -26.505132 | 762.213142 | 870.180384 | 20.330281 | 0.012019 | 5.789669 | 338.973694 | 2.0 | F850LP | 1.674510e+09 | iexn02020 | 2.0 | 02 | 716.0 | 98.681168 | 143.685147 |
| 6 | J160644.67-203342.8 | iexn03010_drc.fits | F814W | 241.686092 | -20.561958 | 765.630228 | 867.979551 | 21.493397 | 0.011936 | 3.114166 | 387.477783 | 3.0 | F814W | 1.674464e+09 | iexn03010 | 2.0 | 03 | 712.0 | 101.059700 | 146.062551 |
| 7 | J160644.67-203342.8 | iexn03020_drc.fits | F850LP | 241.686094 | -20.561958 | 765.408206 | 868.467341 | 20.318072 | 0.012070 | 3.011052 | 357.040802 | 3.0 | F850LP | 1.674465e+09 | iexn03020 | 2.0 | 03 | 712.0 | 101.059700 | 146.062551 |
| 8 | J161434.76-241933.4 | iexn04010_drc.fits | F814W | 243.644804 | -24.326008 | 767.251059 | 867.473929 | 21.553108 | 0.011095 | 2.634757 | 406.735779 | 4.0 | F814W | 1.678931e+09 | iexn04010 | 2.0 | 04 | 712.0 | 99.990402 | 144.993916 |
| 9 | J161434.76-241933.4 | iexn04020_drc.fits | F850LP | 243.644805 | -24.326008 | 767.009600 | 867.939866 | 20.299904 | 0.011329 | 2.285835 | 361.328217 | 4.0 | F850LP | 1.678932e+09 | iexn04020 | 2.0 | 04 | 712.0 | 99.990402 | 144.993916 |
| 10 | J162810.30-264024.2 | iexn05010_drc.fits | F814W | 247.042901 | -26.673437 | 766.601507 | 868.267308 | 21.419805 | 0.010367 | 2.226657 | 451.694641 | 5.0 | F814W | 1.679079e+09 | iexn05010 | 2.0 | 05 | 716.0 | 100.418800 | 145.422670 |
| 11 | J162810.30-264024.2 | iexn05020_drc.fits | F850LP | 247.042902 | -26.673437 | 766.336710 | 868.601284 | 20.159706 | 0.010190 | 2.173232 | 401.216766 | 5.0 | F850LP | 1.679081e+09 | iexn05020 | 2.0 | 05 | 716.0 | 100.418800 | 145.422670 |
| 12 | J160731.61-214654.6 | iexn06010_drc.fits | F814W | 241.881676 | -21.781889 | 767.081804 | 866.901849 | 22.182586 | 0.022100 | 3.231350 | 215.942444 | 6.0 | F814W | 1.674515e+09 | iexn06010 | 2.0 | 06 | 712.0 | 100.292198 | 145.295285 |
| 13 | J160731.61-214654.6 | iexn06020_drc.fits | F850LP | 241.881676 | -21.781889 | 766.867054 | 867.166727 | 20.929665 | 0.021179 | 3.051393 | 187.888596 | 6.0 | F850LP | 1.674516e+09 | iexn06020 | 2.0 | 06 | 712.0 | 100.292198 | 145.295285 |
| 14 | J155543.75-232028.9 | iexn07010_drc.fits | F814W | 238.932269 | -23.341438 | 765.825833 | 868.625604 | 21.854202 | 0.015996 | 2.572152 | 284.837280 | 7.0 | F814W | 1.674520e+09 | iexn07010 | 2.0 | 07 | 712.0 | 100.274902 | 145.278236 |
| 15 | J155543.75-232028.9 | iexn07020_drc.fits | F850LP | 238.932269 | -23.341438 | 765.608929 | 869.098238 | 20.710080 | 0.016869 | 2.555464 | 252.469177 | 7.0 | F850LP | 1.674522e+09 | iexn07020 | 2.0 | 07 | 712.0 | 100.274902 | 145.278236 |
| 16 | J154915.32-244139.1 | iexn08010_drc.fits | F814W | 237.313809 | -24.694252 | 765.740929 | 868.295724 | 22.007757 | 0.017982 | 2.829834 | 249.846924 | 8.0 | F814W | 1.674526e+09 | iexn08010 | 2.0 | 08 | 712.0 | 100.098801 | 145.102367 |
| 17 | J154915.32-244139.1 | iexn08020_drc.fits | F850LP | 237.313810 | -24.694252 | 765.619928 | 868.761725 | 20.915487 | 0.020411 | 2.682921 | 199.514282 | 8.0 | F850LP | 1.674528e+09 | iexn08020 | 2.0 | 08 | 712.0 | 100.098801 | 145.102367 |
| 18 | J160918.67-222923.9 | iexn09010_drc.fits | F814W | 242.327752 | -22.490052 | 762.743761 | 866.727766 | 22.001171 | 0.018724 | 6.191572 | 242.439774 | 9.0 | F814W | 1.674423e+09 | iexn09010 | 2.0 | 09 | 712.0 | 99.707359 | 144.710595 |
| 19 | J160918.67-222923.9 | iexn09020_drc.fits | F850LP | 242.327752 | -22.490052 | 762.620949 | 866.796443 | 20.883590 | 0.020491 | 6.260748 | 191.753143 | 9.0 | F850LP | 1.674425e+09 | iexn09020 | 2.0 | 09 | 712.0 | 99.707359 | 144.710595 |
| 20 | J160656.33-165644.6 | iexn10010_drc.fits | F814W | 241.734689 | -16.945773 | 767.150769 | 866.806861 | 22.110921 | 0.020702 | 3.304138 | 217.606400 | 10.0 | F814W | 1.674429e+09 | iexn10010 | 2.0 | 10 | 712.0 | 103.192200 | 148.194409 |
| 21 | J160656.33-165644.6 | iexn10020_drc.fits | F850LP | 241.734691 | -16.945775 | 766.981161 | 867.158018 | 20.930442 | 0.021017 | 3.019088 | 201.309143 | 10.0 | F850LP | 1.674431e+09 | iexn10020 | 2.0 | 10 | 712.0 | 103.192200 | 148.194409 |
| 22 | J042705.86+261520.3 | iexn54010_drc.fits | F814W | 66.774431 | 26.255547 | 775.939388 | 859.286880 | 21.223795 | 0.008219 | 13.334347 | 413.132355 | 11.0 | F814W | 1.704994e+09 | iexn54010 | 2.0 | 54 | 712.0 | 266.034393 | -48.960802 |
| 23 | J042705.86+261520.3 | iexn54020_drc.fits | F850LP | 66.774431 | 26.255554 | 775.359033 | 859.690031 | 20.284122 | 0.010910 | 12.666919 | 302.929840 | 11.0 | F850LP | 1.704996e+09 | iexn54020 | 2.0 | 54 | 712.0 | 266.034393 | -48.960802 |
Now we assemble the mvs_dataframe populating the expected columns with the values from the entries of the mvs_input_target_df
[7]:
pos=0
mvs_dataframe=pd.DataFrame(columns=['unq_ids','target','ext', 'visit','x_f814w','y_f814w','x_f850lp','y_f850lp','fitsroot_f814w','fitsroot_f850lp', 'exptime_f814w','exptime_f850lp', 'pav3_f814w', 'pav3_f850lp', 'rota_f814w', 'rota_f850lp','flag_f814w','flag_f850lp'])
mvs_dataframe['flag_f814w']='rejected'
mvs_dataframe['flag_f850lp']='rejected'
for id in tqdm(mvs_input_target_df.unq_ids.unique()):
target=mvs_input_target_df.loc[(mvs_input_target_df.unq_ids==id),'target'].values[0]
df_F814W = mvs_input_target_df.loc[(mvs_input_target_df.unq_ids==id)&(mvs_input_target_df.filters=='F814W')].sort_values(['unittime'])
df_F850LP = mvs_input_target_df.loc[(mvs_input_target_df.unq_ids==id)&(mvs_input_target_df.filters=='F850LP')].sort_values(['unittime'])
num = df_F814W.unq_ids.count() if df_F814W.unq_ids.count() >= df_F850LP.unq_ids.count() else df_F850LP.unq_ids.count()
for elno in range(num):
mvs_dataframe.loc[pos,'unq_ids'] = id
if not df_F814W.empty and elno <= df_F814W.unq_ids.count()-1:
mvs_dataframe.loc[pos,'target'] = df_F814W.target.unique()
mvs_dataframe.loc[pos,'ext'] = 1 if df_F814W.ccd.unique() == 2 else 4
mvs_dataframe.loc[pos,'visit'] = df_F814W.visit.unique()
mvs_dataframe.loc[pos,['x_f814w','y_f814w','fitsroot_f814w','exptime_f814w','pav3_f814w','rota_f814w']] =df_F814W.iloc[elno][['x','y','fitsroot','exptime','pav3','rota']].values
else:
mvs_dataframe.loc[pos,'target'] = df_F850LP.target.unique()
mvs_dataframe.loc[pos,'ext'] = 1 if df_F850LP.ccd.unique() == 2 else 4
mvs_dataframe.loc[pos,'visit'] = df_F850LP.visit.unique()
if not df_F850LP.empty and elno <= df_F850LP.unq_ids.count()-1:
mvs_dataframe.loc[pos,['x_f850lp','y_f850lp','fitsroot_f850lp','exptime_f850lp','pav3_f850lp','rota_f850lp']] =df_F850LP.iloc[elno][['x','y','fitsroot','exptime','pav3','rota']].values
pos+=1
mvs_dataframe
100%|██████████| 12/12 [00:00<00:00, 345.74it/s]
[7]:
| unq_ids | target | ext | visit | x_f814w | y_f814w | x_f850lp | y_f850lp | fitsroot_f814w | fitsroot_f850lp | exptime_f814w | exptime_f850lp | pav3_f814w | pav3_f850lp | rota_f814w | rota_f850lp | flag_f814w | flag_f850lp | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0.0 | J155150.21-213457.4 | 1 | 13 | 766.682062 | 870.519962 | 766.297865 | 870.86323 | iexn13010 | iexn13020 | 712.0 | 712.0 | 124.436996 | 124.436996 | 169.438303 | 169.438303 | NaN | NaN |
| 1 | 1.0 | J164636.12-231337.6 | 1 | 01 | 769.100793 | 866.146099 | 768.870891 | 866.488792 | iexn01010 | iexn01020 | 712.0 | 712.0 | 97.360497 | 97.360497 | 142.364009 | 142.364009 | NaN | NaN |
| 2 | 2.0 | J155416.68-263018.1 | 1 | 02 | 762.415167 | 869.798303 | 762.213142 | 870.180384 | iexn02010 | iexn02020 | 716.0 | 716.0 | 98.681168 | 98.681168 | 143.685147 | 143.685147 | NaN | NaN |
| 3 | 3.0 | J160644.67-203342.8 | 1 | 03 | 765.630228 | 867.979551 | 765.408206 | 868.467341 | iexn03010 | iexn03020 | 712.0 | 712.0 | 101.0597 | 101.0597 | 146.062551 | 146.062551 | NaN | NaN |
| 4 | 4.0 | J161434.76-241933.4 | 1 | 04 | 767.251059 | 867.473929 | 767.0096 | 867.939866 | iexn04010 | iexn04020 | 712.0 | 712.0 | 99.990402 | 99.990402 | 144.993916 | 144.993916 | NaN | NaN |
| 5 | 5.0 | J162810.30-264024.2 | 1 | 05 | 766.601507 | 868.267308 | 766.33671 | 868.601284 | iexn05010 | iexn05020 | 716.0 | 716.0 | 100.4188 | 100.4188 | 145.42267 | 145.42267 | NaN | NaN |
| 6 | 6.0 | J160731.61-214654.6 | 1 | 06 | 767.081804 | 866.901849 | 766.867054 | 867.166727 | iexn06010 | iexn06020 | 712.0 | 712.0 | 100.292198 | 100.292198 | 145.295285 | 145.295285 | NaN | NaN |
| 7 | 7.0 | J155543.75-232028.9 | 1 | 07 | 765.825833 | 868.625604 | 765.608929 | 869.098238 | iexn07010 | iexn07020 | 712.0 | 712.0 | 100.274902 | 100.274902 | 145.278236 | 145.278236 | NaN | NaN |
| 8 | 8.0 | J154915.32-244139.1 | 1 | 08 | 765.740929 | 868.295724 | 765.619928 | 868.761725 | iexn08010 | iexn08020 | 712.0 | 712.0 | 100.098801 | 100.098801 | 145.102367 | 145.102367 | NaN | NaN |
| 9 | 9.0 | J160918.67-222923.9 | 1 | 09 | 762.743761 | 866.727766 | 762.620949 | 866.796443 | iexn09010 | iexn09020 | 712.0 | 712.0 | 99.707359 | 99.707359 | 144.710595 | 144.710595 | NaN | NaN |
| 10 | 10.0 | J160656.33-165644.6 | 1 | 10 | 767.150769 | 866.806861 | 766.981161 | 867.158018 | iexn10010 | iexn10020 | 712.0 | 712.0 | 103.1922 | 103.1922 | 148.194409 | 148.194409 | NaN | NaN |
| 11 | 11.0 | J042705.86+261520.3 | 1 | 54 | 775.939388 | 859.28688 | 775.359033 | 859.690031 | iexn54010 | iexn54020 | 712.0 | 712.0 | 266.034393 | 266.034393 | -48.960802 | -48.960802 | NaN | NaN |
For the mvs_dataframe, we also add a new index mvs_ids that assign a unique values to each entry in the mvs_dataframe. Since we have one visit for each targets, the mvs_ids and the unq_ids are same, but if we had multiple visits then for each target, then for each unq_ids we would expect a multiple entries in the mvs_dataframe, each with its own specific mvs_ids.
[8]:
mvs_dataframe = mvs_dataframe.reset_index().rename(columns={'index':'mvs_ids'})
mvs_dataframe['unq_ids']=mvs_dataframe.unq_ids.astype(int)
mvs_dataframe
[8]:
| mvs_ids | unq_ids | target | ext | visit | x_f814w | y_f814w | x_f850lp | y_f850lp | fitsroot_f814w | fitsroot_f850lp | exptime_f814w | exptime_f850lp | pav3_f814w | pav3_f850lp | rota_f814w | rota_f850lp | flag_f814w | flag_f850lp | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | J155150.21-213457.4 | 1 | 13 | 766.682062 | 870.519962 | 766.297865 | 870.86323 | iexn13010 | iexn13020 | 712.0 | 712.0 | 124.436996 | 124.436996 | 169.438303 | 169.438303 | NaN | NaN |
| 1 | 1 | 1 | J164636.12-231337.6 | 1 | 01 | 769.100793 | 866.146099 | 768.870891 | 866.488792 | iexn01010 | iexn01020 | 712.0 | 712.0 | 97.360497 | 97.360497 | 142.364009 | 142.364009 | NaN | NaN |
| 2 | 2 | 2 | J155416.68-263018.1 | 1 | 02 | 762.415167 | 869.798303 | 762.213142 | 870.180384 | iexn02010 | iexn02020 | 716.0 | 716.0 | 98.681168 | 98.681168 | 143.685147 | 143.685147 | NaN | NaN |
| 3 | 3 | 3 | J160644.67-203342.8 | 1 | 03 | 765.630228 | 867.979551 | 765.408206 | 868.467341 | iexn03010 | iexn03020 | 712.0 | 712.0 | 101.0597 | 101.0597 | 146.062551 | 146.062551 | NaN | NaN |
| 4 | 4 | 4 | J161434.76-241933.4 | 1 | 04 | 767.251059 | 867.473929 | 767.0096 | 867.939866 | iexn04010 | iexn04020 | 712.0 | 712.0 | 99.990402 | 99.990402 | 144.993916 | 144.993916 | NaN | NaN |
| 5 | 5 | 5 | J162810.30-264024.2 | 1 | 05 | 766.601507 | 868.267308 | 766.33671 | 868.601284 | iexn05010 | iexn05020 | 716.0 | 716.0 | 100.4188 | 100.4188 | 145.42267 | 145.42267 | NaN | NaN |
| 6 | 6 | 6 | J160731.61-214654.6 | 1 | 06 | 767.081804 | 866.901849 | 766.867054 | 867.166727 | iexn06010 | iexn06020 | 712.0 | 712.0 | 100.292198 | 100.292198 | 145.295285 | 145.295285 | NaN | NaN |
| 7 | 7 | 7 | J155543.75-232028.9 | 1 | 07 | 765.825833 | 868.625604 | 765.608929 | 869.098238 | iexn07010 | iexn07020 | 712.0 | 712.0 | 100.274902 | 100.274902 | 145.278236 | 145.278236 | NaN | NaN |
| 8 | 8 | 8 | J154915.32-244139.1 | 1 | 08 | 765.740929 | 868.295724 | 765.619928 | 868.761725 | iexn08010 | iexn08020 | 712.0 | 712.0 | 100.098801 | 100.098801 | 145.102367 | 145.102367 | NaN | NaN |
| 9 | 9 | 9 | J160918.67-222923.9 | 1 | 09 | 762.743761 | 866.727766 | 762.620949 | 866.796443 | iexn09010 | iexn09020 | 712.0 | 712.0 | 99.707359 | 99.707359 | 144.710595 | 144.710595 | NaN | NaN |
| 10 | 10 | 10 | J160656.33-165644.6 | 1 | 10 | 767.150769 | 866.806861 | 766.981161 | 867.158018 | iexn10010 | iexn10020 | 712.0 | 712.0 | 103.1922 | 103.1922 | 148.194409 | 148.194409 | NaN | NaN |
| 11 | 11 | 11 | J042705.86+261520.3 | 1 | 54 | 775.939388 | 859.28688 | 775.359033 | 859.690031 | iexn54010 | iexn54020 | 712.0 | 712.0 | 266.034393 | 266.034393 | -48.960802 | -48.960802 | NaN | NaN |
Now we want to populate the flag_<filter> entry in the catalog. When the small_tiles command is run, it will generate an image with a small tile for each visit/targets in the mvs_dataframe. The default will put all the sources as good_psf reference. The images can be visually inspected to provide a list of ids to flag either as good_targets (target for the PSF subtraction but not included in the PSF library), rejected or known_double to be ignored by the pipeline
[9]:
path2tiles='/Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/data/small_tiles'
bad_F814W=[]
kd_F814W=[]
good_F814W=[]
bad_F850LP=[]
kd_F850LP=[]
good_F850LP=[]
dict={'bad_f814w':bad_F814W,'kd_f814w':kd_F814W,'bad_f850lp':bad_F850LP,'kd_f850lp':kd_F850LP,'good_f814w':good_F814W,'good_f850lp':good_F850LP,}
mvs_dataframe = small_tiles(mvs_dataframe,path2fits, path2tiles, ['f814w','f850lp'], dict=dict,nrows=8, ncols=8,ext='_drc')
mvs_dataframe
[9]:
| mvs_ids | unq_ids | target | ext | visit | x_f814w | y_f814w | x_f850lp | y_f850lp | fitsroot_f814w | fitsroot_f850lp | exptime_f814w | exptime_f850lp | pav3_f814w | pav3_f850lp | rota_f814w | rota_f850lp | flag_f814w | flag_f850lp | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | J155150.21-213457.4 | 1 | 13 | 766.682062 | 870.519962 | 766.297865 | 870.86323 | iexn13010 | iexn13020 | 712.0 | 712.0 | 124.436996 | 124.436996 | 169.438303 | 169.438303 | good_psf | good_psf |
| 1 | 1 | 1 | J164636.12-231337.6 | 1 | 01 | 769.100793 | 866.146099 | 768.870891 | 866.488792 | iexn01010 | iexn01020 | 712.0 | 712.0 | 97.360497 | 97.360497 | 142.364009 | 142.364009 | good_psf | good_psf |
| 2 | 2 | 2 | J155416.68-263018.1 | 1 | 02 | 762.415167 | 869.798303 | 762.213142 | 870.180384 | iexn02010 | iexn02020 | 716.0 | 716.0 | 98.681168 | 98.681168 | 143.685147 | 143.685147 | good_psf | good_psf |
| 3 | 3 | 3 | J160644.67-203342.8 | 1 | 03 | 765.630228 | 867.979551 | 765.408206 | 868.467341 | iexn03010 | iexn03020 | 712.0 | 712.0 | 101.0597 | 101.0597 | 146.062551 | 146.062551 | good_psf | good_psf |
| 4 | 4 | 4 | J161434.76-241933.4 | 1 | 04 | 767.251059 | 867.473929 | 767.0096 | 867.939866 | iexn04010 | iexn04020 | 712.0 | 712.0 | 99.990402 | 99.990402 | 144.993916 | 144.993916 | good_psf | good_psf |
| 5 | 5 | 5 | J162810.30-264024.2 | 1 | 05 | 766.601507 | 868.267308 | 766.33671 | 868.601284 | iexn05010 | iexn05020 | 716.0 | 716.0 | 100.4188 | 100.4188 | 145.42267 | 145.42267 | good_psf | good_psf |
| 6 | 6 | 6 | J160731.61-214654.6 | 1 | 06 | 767.081804 | 866.901849 | 766.867054 | 867.166727 | iexn06010 | iexn06020 | 712.0 | 712.0 | 100.292198 | 100.292198 | 145.295285 | 145.295285 | good_psf | good_psf |
| 7 | 7 | 7 | J155543.75-232028.9 | 1 | 07 | 765.825833 | 868.625604 | 765.608929 | 869.098238 | iexn07010 | iexn07020 | 712.0 | 712.0 | 100.274902 | 100.274902 | 145.278236 | 145.278236 | good_psf | good_psf |
| 8 | 8 | 8 | J154915.32-244139.1 | 1 | 08 | 765.740929 | 868.295724 | 765.619928 | 868.761725 | iexn08010 | iexn08020 | 712.0 | 712.0 | 100.098801 | 100.098801 | 145.102367 | 145.102367 | good_psf | good_psf |
| 9 | 9 | 9 | J160918.67-222923.9 | 1 | 09 | 762.743761 | 866.727766 | 762.620949 | 866.796443 | iexn09010 | iexn09020 | 712.0 | 712.0 | 99.707359 | 99.707359 | 144.710595 | 144.710595 | good_psf | good_psf |
| 10 | 10 | 10 | J160656.33-165644.6 | 1 | 10 | 767.150769 | 866.806861 | 766.981161 | 867.158018 | iexn10010 | iexn10020 | 712.0 | 712.0 | 103.1922 | 103.1922 | 148.194409 | 148.194409 | good_psf | good_psf |
| 11 | 11 | 11 | J042705.86+261520.3 | 1 | 54 | 775.939388 | 859.28688 | 775.359033 | 859.690031 | iexn54010 | iexn54020 | 712.0 | 712.0 | 266.034393 | 266.034393 | -48.960802 | -48.960802 | good_psf | good_psf |
As a sanity check, we assign the flag rejected also to any entry with nan as coordiantes
[10]:
mvs_dataframe.loc[mvs_dataframe['flag_f814w'].isna(),'flag_f814w'] = 'rejected'
mvs_dataframe.loc[mvs_dataframe['flag_f850lp'].isna(),'flag_f850lp'] = 'rejected'
mvs_dataframe
[10]:
| mvs_ids | unq_ids | target | ext | visit | x_f814w | y_f814w | x_f850lp | y_f850lp | fitsroot_f814w | fitsroot_f850lp | exptime_f814w | exptime_f850lp | pav3_f814w | pav3_f850lp | rota_f814w | rota_f850lp | flag_f814w | flag_f850lp | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | J155150.21-213457.4 | 1 | 13 | 766.682062 | 870.519962 | 766.297865 | 870.86323 | iexn13010 | iexn13020 | 712.0 | 712.0 | 124.436996 | 124.436996 | 169.438303 | 169.438303 | good_psf | good_psf |
| 1 | 1 | 1 | J164636.12-231337.6 | 1 | 01 | 769.100793 | 866.146099 | 768.870891 | 866.488792 | iexn01010 | iexn01020 | 712.0 | 712.0 | 97.360497 | 97.360497 | 142.364009 | 142.364009 | good_psf | good_psf |
| 2 | 2 | 2 | J155416.68-263018.1 | 1 | 02 | 762.415167 | 869.798303 | 762.213142 | 870.180384 | iexn02010 | iexn02020 | 716.0 | 716.0 | 98.681168 | 98.681168 | 143.685147 | 143.685147 | good_psf | good_psf |
| 3 | 3 | 3 | J160644.67-203342.8 | 1 | 03 | 765.630228 | 867.979551 | 765.408206 | 868.467341 | iexn03010 | iexn03020 | 712.0 | 712.0 | 101.0597 | 101.0597 | 146.062551 | 146.062551 | good_psf | good_psf |
| 4 | 4 | 4 | J161434.76-241933.4 | 1 | 04 | 767.251059 | 867.473929 | 767.0096 | 867.939866 | iexn04010 | iexn04020 | 712.0 | 712.0 | 99.990402 | 99.990402 | 144.993916 | 144.993916 | good_psf | good_psf |
| 5 | 5 | 5 | J162810.30-264024.2 | 1 | 05 | 766.601507 | 868.267308 | 766.33671 | 868.601284 | iexn05010 | iexn05020 | 716.0 | 716.0 | 100.4188 | 100.4188 | 145.42267 | 145.42267 | good_psf | good_psf |
| 6 | 6 | 6 | J160731.61-214654.6 | 1 | 06 | 767.081804 | 866.901849 | 766.867054 | 867.166727 | iexn06010 | iexn06020 | 712.0 | 712.0 | 100.292198 | 100.292198 | 145.295285 | 145.295285 | good_psf | good_psf |
| 7 | 7 | 7 | J155543.75-232028.9 | 1 | 07 | 765.825833 | 868.625604 | 765.608929 | 869.098238 | iexn07010 | iexn07020 | 712.0 | 712.0 | 100.274902 | 100.274902 | 145.278236 | 145.278236 | good_psf | good_psf |
| 8 | 8 | 8 | J154915.32-244139.1 | 1 | 08 | 765.740929 | 868.295724 | 765.619928 | 868.761725 | iexn08010 | iexn08020 | 712.0 | 712.0 | 100.098801 | 100.098801 | 145.102367 | 145.102367 | good_psf | good_psf |
| 9 | 9 | 9 | J160918.67-222923.9 | 1 | 09 | 762.743761 | 866.727766 | 762.620949 | 866.796443 | iexn09010 | iexn09020 | 712.0 | 712.0 | 99.707359 | 99.707359 | 144.710595 | 144.710595 | good_psf | good_psf |
| 10 | 10 | 10 | J160656.33-165644.6 | 1 | 10 | 767.150769 | 866.806861 | 766.981161 | 867.158018 | iexn10010 | iexn10020 | 712.0 | 712.0 | 103.1922 | 103.1922 | 148.194409 | 148.194409 | good_psf | good_psf |
| 11 | 11 | 11 | J042705.86+261520.3 | 1 | 54 | 775.939388 | 859.28688 | 775.359033 | 859.690031 | iexn54010 | iexn54020 | 712.0 | 712.0 | 266.034393 | 266.034393 | -48.960802 | -48.960802 | good_psf | good_psf |
Finally we save the mvs_dataframe to the disk
[13]:
import os
os.mkdir('/Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/database')
mvs_dataframe.to_csv('/Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/database/FFP_mvs_target_photometry.csv')
[14]:
mvs_dataframe
[14]:
| mvs_ids | unq_ids | target | ext | visit | x_f814w | y_f814w | x_f850lp | y_f850lp | fitsroot_f814w | fitsroot_f850lp | exptime_f814w | exptime_f850lp | pav3_f814w | pav3_f850lp | rota_f814w | rota_f850lp | flag_f814w | flag_f850lp | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | J155150.21-213457.4 | 1 | 13 | 766.682062 | 870.519962 | 766.297865 | 870.86323 | iexn13010 | iexn13020 | 712.0 | 712.0 | 124.436996 | 124.436996 | 169.438303 | 169.438303 | good_psf | good_psf |
| 1 | 1 | 1 | J164636.12-231337.6 | 1 | 01 | 769.100793 | 866.146099 | 768.870891 | 866.488792 | iexn01010 | iexn01020 | 712.0 | 712.0 | 97.360497 | 97.360497 | 142.364009 | 142.364009 | good_psf | good_psf |
| 2 | 2 | 2 | J155416.68-263018.1 | 1 | 02 | 762.415167 | 869.798303 | 762.213142 | 870.180384 | iexn02010 | iexn02020 | 716.0 | 716.0 | 98.681168 | 98.681168 | 143.685147 | 143.685147 | good_psf | good_psf |
| 3 | 3 | 3 | J160644.67-203342.8 | 1 | 03 | 765.630228 | 867.979551 | 765.408206 | 868.467341 | iexn03010 | iexn03020 | 712.0 | 712.0 | 101.0597 | 101.0597 | 146.062551 | 146.062551 | good_psf | good_psf |
| 4 | 4 | 4 | J161434.76-241933.4 | 1 | 04 | 767.251059 | 867.473929 | 767.0096 | 867.939866 | iexn04010 | iexn04020 | 712.0 | 712.0 | 99.990402 | 99.990402 | 144.993916 | 144.993916 | good_psf | good_psf |
| 5 | 5 | 5 | J162810.30-264024.2 | 1 | 05 | 766.601507 | 868.267308 | 766.33671 | 868.601284 | iexn05010 | iexn05020 | 716.0 | 716.0 | 100.4188 | 100.4188 | 145.42267 | 145.42267 | good_psf | good_psf |
| 6 | 6 | 6 | J160731.61-214654.6 | 1 | 06 | 767.081804 | 866.901849 | 766.867054 | 867.166727 | iexn06010 | iexn06020 | 712.0 | 712.0 | 100.292198 | 100.292198 | 145.295285 | 145.295285 | good_psf | good_psf |
| 7 | 7 | 7 | J155543.75-232028.9 | 1 | 07 | 765.825833 | 868.625604 | 765.608929 | 869.098238 | iexn07010 | iexn07020 | 712.0 | 712.0 | 100.274902 | 100.274902 | 145.278236 | 145.278236 | good_psf | good_psf |
| 8 | 8 | 8 | J154915.32-244139.1 | 1 | 08 | 765.740929 | 868.295724 | 765.619928 | 868.761725 | iexn08010 | iexn08020 | 712.0 | 712.0 | 100.098801 | 100.098801 | 145.102367 | 145.102367 | good_psf | good_psf |
| 9 | 9 | 9 | J160918.67-222923.9 | 1 | 09 | 762.743761 | 866.727766 | 762.620949 | 866.796443 | iexn09010 | iexn09020 | 712.0 | 712.0 | 99.707359 | 99.707359 | 144.710595 | 144.710595 | good_psf | good_psf |
| 10 | 10 | 10 | J160656.33-165644.6 | 1 | 10 | 767.150769 | 866.806861 | 766.981161 | 867.158018 | iexn10010 | iexn10020 | 712.0 | 712.0 | 103.1922 | 103.1922 | 148.194409 | 148.194409 | good_psf | good_psf |
| 11 | 11 | 11 | J042705.86+261520.3 | 1 | 54 | 775.939388 | 859.28688 | 775.359033 | 859.690031 | iexn54010 | iexn54020 | 712.0 | 712.0 | 266.034393 | 266.034393 | -48.960802 | -48.960802 | good_psf | good_psf |
Lets populate the `unq_dataframe’
Similar to the mvs_dataframe now we populate the unq_dataframe with the expected columns.
[15]:
pos=0
unq_dataframe=pd.DataFrame(columns=['unq_ids','ra', 'dec','m_f814w','e_f814w','m_f850lp','e_f850lp','type'])
for id in tqdm(unq_input_target_df.unq_ids.unique()):
unq_input_target_df.loc[(unq_input_target_df.unq_ids==id),'target']=unq_input_target_df.loc[unq_input_target_df.unq_ids==id].target.values[0]
df_F814W = unq_input_target_df.loc[(unq_input_target_df.unq_ids==id)&(unq_input_target_df['filter']=='F814W')]
df_F850LP = unq_input_target_df.loc[(unq_input_target_df.unq_ids==id)&(unq_input_target_df['filter']=='F850LP')]
num = df_F814W.unq_ids.count() if df_F814W.unq_ids.count() >= df_F850LP.unq_ids.count() else df_F850LP.unq_ids.count()
for elno in range(num):
unq_dataframe.loc[pos,'unq_ids'] = id
if not df_F814W.empty:
unq_dataframe.loc[pos,'ra'] = df_F814W.ra.unique()
unq_dataframe.loc[pos,'dec'] = df_F814W.dec.unique()
elif not df_F850LP.empty:
unq_dataframe.loc[pos,'ra'] = df_F850LP.ra.unique()
unq_dataframe.loc[pos,'dec'] = df_F850LP.dec.unique()
unq_dataframe.loc[pos,'type'] = 1
if np.all(mvs_dataframe.loc[mvs_dataframe.unq_ids==id,['flag_f814w','flag_f850lp']]=='rejected'):
unq_dataframe.loc[unq_dataframe.unq_ids==id,'type']=0
if not df_F814W.empty and elno <= df_F814W.unq_ids.count()-1:
unq_dataframe.loc[pos,['m_f814w','e_f814w']] =df_F814W.iloc[elno][['mag_aper','e_mag_aper']].values
if not df_F850LP.empty and elno <= df_F850LP.unq_ids.count()-1:
unq_dataframe.loc[pos,['m_f850lp','e_f850lp']] =df_F850LP.iloc[elno][['mag_aper','e_mag_aper']].values
pos+=1
unq_dataframe=unq_dataframe.sort_values('unq_ids')
unq_dataframe['unq_ids']=unq_dataframe.unq_ids.astype(int)
unq_dataframe
100%|██████████| 12/12 [00:00<00:00, 403.33it/s]
[15]:
| unq_ids | ra | dec | m_f814w | e_f814w | m_f850lp | e_f850lp | type | |
|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 237.9591707009825 | -21.582685895526943 | 22.277353 | 0.021258 | 21.078353 | 0.022984 | 1 |
| 1 | 1 | 251.6504980156288 | -23.22715502396358 | 20.840983 | 0.005901 | 19.749264 | 0.006932 | 1 |
| 2 | 2 | 238.5694232156599 | -26.505131228149015 | 21.448087 | 0.010977 | 20.330281 | 0.012019 | 1 |
| 3 | 3 | 241.6860922957023 | -20.561958458050395 | 21.493397 | 0.011936 | 20.318072 | 0.01207 | 1 |
| 4 | 4 | 243.64480424478836 | -24.326007804504044 | 21.553108 | 0.011095 | 20.299904 | 0.011329 | 1 |
| 5 | 5 | 247.04290079840197 | -26.673436687805328 | 21.419805 | 0.010367 | 20.159706 | 0.01019 | 1 |
| 6 | 6 | 241.88167608033177 | -21.781888992957537 | 22.182586 | 0.0221 | 20.929665 | 0.021179 | 1 |
| 7 | 7 | 238.9322686924369 | -23.341438129414332 | 21.854202 | 0.015996 | 20.71008 | 0.016869 | 1 |
| 8 | 8 | 237.3138087651073 | -24.694251812554015 | 22.007757 | 0.017982 | 20.915487 | 0.020411 | 1 |
| 9 | 9 | 242.3277516372519 | -22.49005155708848 | 22.001171 | 0.018724 | 20.88359 | 0.020491 | 1 |
| 10 | 10 | 241.73468914180376 | -16.945772988183773 | 22.110921 | 0.020702 | 20.930442 | 0.021017 | 1 |
| 11 | 11 | 66.77443071618748 | 26.25554724625119 | 21.223795 | 0.008219 | 20.284122 | 0.01091 | 1 |
We now save it to a file
[16]:
unq_dataframe.to_csv('/Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/database/FFP_unq_target_photometry.csv')
We also save a table linking each unq_ids to each series of mvs_ids
[17]:
unq_input_target_df[['unq_ids','target']].to_csv('/Users/gstrampelli/PycharmProjects/StraKLIP_tutorial_test/database/FFP_unq2target_ids.csv')
Now we have the three staring catalogs to feed to the pipeline to start the reduction of the data.
[ ]: