docs(reliability): Avoid non-deterministic expressions in shuffle keys (#62)
- docs(reliability): Avoid non-deterministic expressions in shuffle keys
Explain how rand(), uuid(), floating-point aggregates and other order-dependent expressions used as shuffle keys can silently produce duplicate and lost rows when Spark retries shuffle map tasks, with a worked example and retry-safe alternatives.
- docs(reliability): Address review feedback
- Use non-negative rand() values in the retry example
- Note that Spark 4.2 enables order-independent shuffle checksums by default
Co-authored-by: shoukasg shoukasg@amazon.com
版权所有:中国计算机学会技术支持:开源发展技术委员会
京ICP备13000930号-9
京公网安备 11010802047560号
Amazon EMR on Amazon Best Practices
A best practices guide for submitting spark applications, integration with hive metastore, security, storage options, debugging options and performance considerations..
Return to Live Docs.
License Summary
The documentation is made available under the Creative Commons Attribution-ShareAlike 4.0 International License. See the LICENSE file.
The sample code within this documentation is made available under the MIT-0 license. See the LICENSE-SAMPLECODE file.