Autonomous data pipeline intelligence platform running in production with agents that analyze, optimize, and act on Spark workloads