eduzhai > Applied Sciences > Engineering >

Age-Based Coded Computation for Bias Reduction in Distributed Learning

  • Save

... pages left unread,continue reading

Document pages: 6 pages

Abstract: Coded computation can be used to speed up distributed learning in thepresence of straggling workers. Partial recovery of the gradient vector canfurther reduce the computation time at each iteration; however, this can resultin biased estimators, which may slow down convergence, or even causedivergence. Estimator bias will be particularly prevalent when the stragglingbehavior is correlated over time, which results in the gradient estimatorsbeing dominated by a few fast servers. To mitigate biased estimators, we designa $timely$ dynamic encoding framework for partial recovery that includes anordering operator that changes the codewords and computation orders at workersover time. To regulate the recovery frequencies, we adopt an $age$ metric inthe design of the dynamic encoding scheme. We show through numerical resultsthat the proposed dynamic encoding strategy increases the timeliness of therecovered computations, which as a result, reduces the bias in model updates,and accelerates the convergence compared to the conventional static partialrecovery schemes.

Please select stars to rate!


0 comments Sign in to leave a comment.

    Data loading, please wait...