從雪花到企業級 Apache Spark?.pdf

1、From Snowflake To Enterprise-Scale Apache SparkNic Jansma+Amir SkovronikAkamaiDatabricks2023From Snowflake To Enterprise-Scale Apache SparkNic JSr.Principal Lead Engineer(mPulse)Amir SDistinguished Engineer(Asgard)1_DAIS_Title_SlidemPulse:Real User MonitoringWhat is mPulse?Real User Monitoring(RUM)m

2、Pulse provides real-time user experience and performance analytics,and maps those results to business goals and outcomes.4Scale 2 billion beacons/day(no sampling!)Real-Time(aggregate)dashboards:User experiences are reflected within 5-10s7 TB raw data/dayWaterfall(individual)dashboards:Full debug tra

3、ce of every page load+beacon available within 5 minutes4 TB raw logs/day5Scale13 months retention50 fact/dimension tables1 T rows1 PB storage60 QPS6Goals of MigrationEarly Snowflake adopter but needs have changedHighest cloud cost for mPulse($10m/year)New Akamai internal team(Asgard)dedicated to pro

4、viding a data warehouse solution for all of AkamaimPulse was to be one of the first large products to transition to AsgardUnique technical challengesEqual-or-better performanceCustomers shouldnt notice a difference7ChallengesYears of assumptions built into mPulse from Snowflake dependencySnowflake m

5、ade it easy to“throw$at the problem”by just up-sizing warehouses so we never focused on optimizationNeeded a comprehensive query inventory,and discussions and plans for how to transition each workloadOther internal teams depend on mPulse data,and they need their own migration paths and hand-holdingN

6、ew tooling needsOrganizationally,two sibling teams(mPulse,Asgard)needed to figure out how to work together and support each other8Asgard:Enterprise-ScaleApache Spark9What is Asgard?An homegrown cloud based Data WarehouseSnowflake like deployment model(S/M/L/XL WH)Snowflake like ingest API(COPY INTO)

7、Spark SQL query APISpark SQL API for ETL execution InfrastructureCompute:AKSStorage:Azure Gen210Asgard Secret Sauce Customized&enhanced Spark versionUnique partitioning modelState of the art columnar format 11Customized&Enhanced Spark VersionInternal code optimization Optimize synchronized blocks,da

8、ta structures&SQL functions Improved cached Data frame in memory compressionImproved driver stability protectionCustom Strategy,Rules&Filters in order to enable better push down capabilities Cached data Locality awareness AZ locality awareness 12Unique Partitioning ModelInspired by delta lake.Intern

9、ally a file is split according to table partition keys.A file footer points to the start/end offsets of each partition.Meta service:A custom service which exposes per table metadata on all files in a SQL queryable format.Metadata includes:column value ranges,size,cached location,etcThe data is store

10、d in an in memory database which able to serve X10 q/s at 200ms.The service can be scale out easily to accommodate with high query rate.13State of the art columnar format Code name:PadawanExtend pushdown predicates capabilities(Regex,UDF etc)Extend pushdown aggregation capabilities.Support delete pu

11、shdown.Support“explode”pushdown.Support optimized data encryption In local benchmark compared to Parquet:10-15%storage footprint reduction.Same write time.20-80%improved query time.141_DAIS_Title_SlideMigrationResults15$10m/year savings(80%cost reduction)Better Performance(20-80%)Adaptable for Our N

12、eeds16The fine printMigrations have a cost:Years to completeDeveloper fatigueOpportunity cost vs.other features17Whats next?mPulseFurther infrastructure optimizationsMigration to Akamai Cloud-LinodeOpportunities to build new featuresAsgardMigration to Akamai Cloud-LinodeAuto scaling&self management WHSpark jobs on demandResearch tools 18Thanks!(Q&A)19