Base cores on sparkContext.defaultParallelism

[benstrum]

The code and docs show that the number of threads is limited to 2 if spark.driver.cores is not set. There are often times when this value is not set and thus gets set to 2. Suggestions include the following:

• base the value on sparkContext.defaultParallelism
• or allow the value to be configured during initialization