[SysDeg] Worksharing Framework and its design: Some modifications


Nhat HOANG

sparksql-server-system-design-3

After a lot of discussions, we decided to change the design of the system a little bit, so it can be more general and extensible. The new design is described as the figure above.

The WorkSharing Detector remains the same as the old design. Its goal is generating bags of DAGs which are labeled with sharing-type or no-sharing.

The Cost Model is a new component. It provides an interface so that users can plug their own cost model into the system (User defined cost model).

With a bag from the output of the previous and a cost model associated with its sharing type, the Optimizer component will do the job: pick the plan with the lowest cost.

The best plan will be the input of the Rewriter. The rewriter will know which rule it should apply to the plan to get the transformed plan. The rule is also plugable and…

View original post 238 more words