I ran across an issue in my research and was wondering if any of you had some ideas that could help. I have a set of data that I combine in different ways to obtain certain results. There are a very large number of potential results because of the very large number of combinations (about 10^21). However, I suspect many of the results coming from these combinations are non-unique or very similar to each other. I suspect this because I've randomly computed1000 results and not all 1000 are unique. I want to obtain a much smaller subset of results that is representative of this set of all results. How do I go about obtaining a good size for this subset, or an error estimate for a given size? For example, is there a way to guess the number of unique values in the set of all possible results and use this to compute a probability that my results occurred, given the number of unique values that I found in the sample of 1000? Anything larger than about 1000 will take too long to postprocess.
