07.05

Week 7, Day 5

Improved code organization greatly, adding complete documentation to the closest worlds implementation. Wrote up Roger's test case for evaluating this implementation. Also, found a Euclidean distance transformation which yields values in the range [0, 1] using exponentiation: 1/e^distance. Since I noticed that the similarity scores get very small (>0.0001), I added in a power-scaling mechanism which brings the lowest nonzero similarity score to 0.01.

I began looking for real-world datasets, and found the sources listed below. Testing the counterfactual module with real-world data will be tricky, due to the fundamental problem of causal inference (it's impossible to observe the causal effect on a single unit -- a given person either took the treatment or didn't).

• Kaggle
• Google Dataset Search
  
• AWS Open Data
• Azure Public Datasets
• UCI ML
• CMU Database
• GitHub ML

07.04

Week 7, Day 4

Met with the team this morning and discussed progress. Roger pointed out that the Prob class has an isDiscrete method, which I am now using to split the dataset into continuous and discrete vars in the closest worlds implementation. Now the code steps are as follows:

1. Convert the dataset dictionary to a pandas DataFrame.
2. Conditionalize on the counterfactual and filter columns on the conditional clause.
3. Calculate k (how many worlds to find; hard-coded at 1% of dataset length).
4. Split the dataset into two dataframes for continuous and discrete columns.
5. Use Jaccard coefficient to compare discrete dataset against discrete RVs of the given unit observation.
6. Use Euclidean distance* to compare continuous dataset against continuous RVs of the given unit observation.
7. Use the above values to produce a weighted-average similarity score.
8. Sort the records by similarity, and restrict the dataset to the k closest.
9. Return a call to Prob.predictDist, targeting the RV and using only the k closest records as data.

* See Week 7, Day 3 -- I realized these limitations also apply to the continuous variables, as cos sim breaks for colinear variables: [1 1] = [9 9]. Thus, I changed the similarity metric to Euclidean distance; however, this doesn't produce scores in the range [0, 1]; TBD tomorrow.

07.03

Week 7, Day 3

After testing, I've realized that cosine similarity is not the best metric for a mix of continuous and discrete variables.  In particular, cosine similarity doesn't work for binary variables because cosine similarity does not account for zero vectors. these are an inevitability with conditionalization sets comprised only of binary variables. For example, if you have three binary variables you want to compare, you'll have the combination [0, 0, 0]

with a magnitude of zero, and since that's in the denominator of the cos sim function you'll get compilation errors every time. Moreover, cosine similarity is based on a normalized dot product. this means that the cosine similarity of 0 and 0 is the same as the similarity between 0 and 1, when it should be the same as 1 and 1. And conceptually, this representation doesn't capture the behavior of categorical variables, since two mutually exclusive categories shouldn't be placed on the same axis, they should be orthogonal. Instead of treating 0 and 1 as numbers, they should be treated as vectors on their own axes.

After research, I found the solution of using matching coefficients for categorical RVs, particularly using the Jaccard coefficient metric. This means we need a way of knowing which RVs are discrete. which I will bring up in tomorrow's team meeting.

07.02

Week 7, Day 2

• Finished closest worlds implementation, using cosine similarity to compare each record (row) in the dataset against the given unit observation. 
• Decided to filter on the counterfactual early on, to ensure that the number of closest worlds found is not reduced at the end.
• Updated the variable data types to be consistent with the existing codebase, and added PyDoc.

07.01

Week 7, Day 1

Read Pearl's paper about the "effect of treatment on the treated" (source); this may be a useful tool a bit later on, as it allows estimation of a counterfactual distribution under conditional intervention.

Started implementing the "closest worlds" approach, using DataFrames to easily represent and transform each observation in the dataset.

Met with Roger and discussed reordered to-do list:

1. Implement Lewis' imaging approach (closest worlds).
2. Implement test cases.
3. Search for real-world datasets.
4. Implement Pearl's deterministic/probabilistic approaches.
5. Implement ETT.
6. Implement mediation/additive intervention.
7. Implement PNS.