PSID Folder Documentation

>>> NOTE: This directory has the intermediate programs I used to create the PSID data for
    Dynan (BPEA, 2012). But, it doesn't have any of the data or output files for space
    reasons. If you want to regenerate the data set from scratch, do the steps described
    below. Karen Dynan June 2012.

The programs in this folder extract and do basic processing of data downloaded from the PSID website.
The output is a set of stata datafiles, each corresponding to a different piece of information (e.g. head
age). There is one observation for every person who has ever headed a household in the PSID. There are 
variables, then, for each wave of the PSID. For example, hage77 corresponds to the age of the head of 
the household that person was in for the 1977 wave (note that the person was a head at some point, but
not necessarily head in 1977).

These datasets are stored in the pro_w_data directory. You can merge them together by merging on 
persid.

To run the programs:

1. If you are going to add a new year to the PSID, you'll need to download the data from the PSID
   website (if the data revise for older years, you'll need to do this as well, but this doesn't
   seem to happen much).

   >> Download the data to the psid_data/Download subdirectory. It will be in zipped form. If you are
      adding a new year, you'll want to download both the new "FAMyyyyER" zip file and the "indyyyyer"
      zip file.
   >> Open the FAMyyyyER file and copy the FAMyyyyER.dat file and the corresponding *.do and *.sas files
      to psid_data/Family subdirectory. Then zip up FAMyyyyER.dat. I know this step seemd annoyingly 
      un-automated but if you 'unzipfile' the full archive from Stata, you end up with 6-7 files in your 
      working directory, which you would need to erase, which would be even more annoying.
   >> Open the indyyyyer zip file and do essentially the same thing, copying INDyyyyER.dat to the 
      psid_data/Individual subdirectory and zipping it back up.

2. If you are adding a new year or new variables, you'll need to augment this spreadsheet, which maps
   the PSID mnemonics into the variable names:

   psidvar.xls

   Note also that there is a temporary 2011 permutation on the spreadsheet:

     psidvar_2011_wave.xls: includes the preliminary 2011 variables (which are not yet fully incorporated into
     the system). The 2011 data are in such different format, that they get treated differently in my system,
     but they will eventually be in the same format (expected spring 2013) and at that point we should work 
     them into this system in a fully consistent way.

3. If you are adding a new year or new variables, you'll also need to alter these programs:

   >> readind.sas (only if new year added)

      .. new top part (infix and label coding) taken largely from the sample *.do program provided 
         by the PSID
      .. you'll also need to add variables in the "keep" statement and the "ren" statements to correpond 
         to the new years' info

   >> readfamyy.sas

     .. You'll need to create a new one if you are adding a new year. Model it on the old ones and
        grab the info about what's in the data set (labels, formats, lcrecl, and position) from the  
        sample SAS program provided by the PSID staff. 

     .. The info after the "put" statement comes from the psidvar.xls spreadsheet. It's all the variables
        you want to keep. If you are just adding variables, this is the only part of the programs you need
        to change

   >> labrenyy.do
 
    .. You'll need to create a new one if you are adding a year. As with the sas program, model it on the
       previous year, grabbing info as needed fromm the .do file provided by the PSID staff.

    .. The most important thing is that the variables listed after the 'infile' statement exactly match
       those "put" out by the SAS program. If you are adding variables, you need to change this part.

    .. If you are adding variables, you'll also need to add them as part of the sections of programs that 
       do the renaming. Again, this info comes from the psidvar.xls spreadsheet.

3. Go to the driver program, _read_psid.do

   >> Set the flags for which parts of the extraction you want to do. You'll need to 
      do all parts if you are creating the data set anew.

   >> The first section unzips the yearly PSID Individual files from psid_data/Download, cleans them up, 
      and saves them into /ind_datasets. The variables that are saved are individual’s ID variables.

      read_ind.do 
      This do-file pulls data, labels it, drop variables we don’t need, organizes the rest, and saves our data 
      in the /ind_datasets folder.

      ==> Need to run this if you are adding a new year

   >> The second section unzips the yearly PSID Family and corresponding files from psid_data/Download, 
     combines them with files from /ind_datasets, cleans them up, and saves them into the yr_datasets folder.  
  
     .. readfamXX.sas: reads the data in and then outputs the variables we want to a flat file in the 'raw'
        subdirectory.    
     .. labrenXX.do: reads the flat file into a Stata datasets and renames the variables such they 
        have our mnemonics.

     ==> Need to run this if you are adding a new year or adding new variables

  >> The third section creates a wide dataset, where files are arranged by variables rather than years. 

     Makewide.do
     This do-file creates new wide variables by taking data from different years, combining them, and 
     breaking them up by variable. The results are saved in the /raw_w_datasets folder.

     ==> Need to run if you are adding a new year. If you are just adding new variables, you 
         can run it just for those variables.

  >> The fourth section is temporary code that pulls in the preliminary 2011 data. We can get rid of 
     it once the full data are released.

     ==> Need to run if you are adding new 2011 variables.

  >> The final section processes the wide dataset by calling process.do. 

     process.do
     This do-file processes the wide dataset. This is where variables get transformed, recoded for
     consistency, and combined into new variables, which are stored in the pro_w_datasets directory.

     ==> Need to run if you are adding new variables or new year.
