語言模型：預測、可解釋性及其他.pdf

《語言模型：預測、可解釋性及其他.pdf》由會員分享，可在線閱讀，更多相關《語言模型：預測、可解釋性及其他.pdf（51頁珍藏版）》請在三個皮匠報告上搜索。

1、Transformer Based Time-Series Language Model for Energy ForecastingArtificial Intelligence(AI)and Digital Transformation(DX)Electric Power SummitDr.Satish Natti&Dr.Parul Arora01/07/2025www.gridaxon.io 1Contents Introduction Model Framework Case Study Dataset Description Results Comparison Conclusion

2、1/7/25www.gridaxon.io 2Introduction1/7/25www.gridaxon.io 3CustomersProject DevelopersIPPsPower MarketersISOs/UtilitiesService AreasSite IdentificationInterconnection ProcessGrid ModernizationNERC ComplianceExperience70+years of Cumulative experience30+GW projects across all marketsSite identificatio

3、n to Structuring PPAs AnalysisFeasibility StudiesSystem Impact StudiesEconomic StudiesModeling PSSe/PSLF/ASPEN/PSCAD/PROMOD/TARAIntroduction1/7/25www.gridaxon.io 4Key Idea Use of language models like T5 and GPT-2 for probabilistic time-series energy forecasting Motivation Time series tasks lack a ge

4、neral-purpose,pretrained framework like those in NLPChallenges Limited availability of high-quality time series datasets compared to text Existing forecasting models require dataset-specific fine-tuning and are resource-intensiveObjective Develop a unified,pretrained framework for time series with z

5、ero-shot and probabilistic forecasting capabilities To develop a model that is valuable in situations where historical time-series data is unavailable or when quick results are needed without extensive pre-trainingIntroduction Need for Energy Forecasting Balance supply and demand Reliable power deli

6、very by balancing generation to consumption Cost optimization Optimal resource allocation to lower the costs Renewable energy integration Manage intermittency and curtailment Energy markets Improve trading efficiency and reduce financial risk1/7/25www.gridaxon.io 5Model Framework A time series is tr

7、ansformed into a sequence of tokens via scaling and quantization Then a language model is trained on these tokens using the cross-entropy loss.Once trained,probabilistic forecasts are obtained by sampling multiple future trajectories given the historical context The models in this work are based on

8、the T5 architecture(Text-To-Text Transfer Transformer)Advantages Minimal architecture changes to existing LLMs Compatible with standard language modeling objectives like cross-entropy loss Comparison Traditional methods:domain-specific tuning.TS-LLM:universal framework with pretraining on large,dive

9、rse datasets1/7/25www.gridaxon.io 6Fig.1-(Left)The input time series is scaled and quantized to obtain a sequence of tokens.(Center)The tokens are fed into a language model which may either be an encoder-decoder or a decoder only model.The model is trained using the cross-entropy loss.(Right)During

10、inference,we autoregressively sample tokens from the model and map them back to numerical values.Multiple trajectories are sampled to obtain a predictive distribution.Case Study:Data Description1/7/25www.gridaxon.io 7 The study analyses six(6)distinct energy sources:Coal,Gas,Gas-Combined Cycle(Gas-C

11、C),Hydro,Solar,and Wind.Data is obtained from publicly available resources().The datasets cover a temporal range from 2019 to 2023,representing energy generation metrics for each fuel type.FuelMean(MWh)Median(MWh)Std.Dev.(MWh)Variance(MWh)Minimum(MWh)Maximum(MWh)Skewness(MWh)Kurtosis(MWh)Wind 2706.5

12、312597.1581464.4722144678.8055.2746725.8640.268-0.848Coal2021.9471989.336591.791350216.746527.6353477.2540.150-0.741Gas809.919501.667825.820681977.85260.1344907.3111.8583.341Gas-CC4343.9654284.1431651.6542727961.930652.1568002.1630.032-0.842Hydro15.90511.69916.148260.7610.000101.8531.5062.646Solar48

13、7.5355.429774.150599308.5680.0003516.4891.7642.313Case Study:Data Description1/7/25www.gridaxon.io 8Case Study:Results1/7/25www.gridaxon.io 9Case Study:Results1/7/25www.gridaxon.io 10Case Study:Results1/7/25www.gridaxon.io 11Case Study:Comparison1/7/25www.gridaxon.io 12Note:Validation Score and Pred

14、iction time lower is the betterConclusions The TS-LLM model can process multiple time-series datasets from different domains simultaneously,without training each time-series individually.TS-LLM offers scalable performance across diverse datasets while maintaining a consistent architecture.With pre-t

15、rained models that enable zero-shot forecasting,TS-LLM delivers accuracy comparable to or better than traditional models,especially for median(P50)forecasts,even with minimal or no fine-tuning.This work represents the first instance of testing various energy fuels data using LLMs,demonstrating the v

16、ersatility of LLMs in energy forecasting applications.1/7/25www.gridaxon.io 13References Alexander Alexandrov,Konstantinos Benidis,Michael Bohlke-Schneider,Valentin Flunkert,Jan Gasthaus,Tim Januschowski,Danielle C Maddix,Syama Rangapuram,David Salinas,Jasper Schulz,et al.GluonTS:Probabilistic and N

17、eural Time Series Modeling in Python.The Journal of Machine Learning Research,21(1):46294634,2020.33 Oliver Borchert,David Salinas,Valentin Flunkert,Tim Januschowski,and Stephan Gnnemann.Multiobjective model selection for time series forecasting.arXiv preprint arXiv:2202.08485,2022.21,43 Nate Gruver

18、,Marc Finzi,Shikai Qiu,and Andrew Gordon Wilson.Large Language Models Are Zero-Shot Time Series Forecasters.In Advances in Neural Information Processing Systems,2023.Rob J Hyndman and George Athanasopoulos.Forecasting:principles and practice.OTexts,2018.1,9 Parul Arora,BK Panigrahi,and PN Suganthan.

19、Shallow neural networks to deep neural networks for probabilistic wind forecasting.In 2021 Inter-national Conference on Computing,Communication,and Intelligent Systems(ICCCIS),pages 377382.IEEE,2021.Parul Arora,Abbas Khosravi,BK Panigrahi,and PN Suganthan.Remodelling state-space prediction with deep

20、 neural networks for probabilistic load forecasting.IEEE Transactions on Emerging Topics in Computational Intelligence,2021.Parul Arora,Himanshu Kumar,and BK Panigrahi.A comparative study for short term wind speed forecasting using statistical and machine learning approaches.In 2018 2nd IEEE Interna

21、tional Conference on Power Electronics,Intelligent Control and Energy Systems(ICPEICES),pages 200205.IEEE,2018.1/7/25www.gridaxon.io 14References Yuqi Nie,Nam H.Nguyen,Phanwadee Sinthong,and Jayant Kalagnanam.A time series is worth 64 words:Long-term forecasting with transformers.In International Co

22、nference on Learning Representations,2023.Alec Radford,Jeffrey Wu,Rewon Child,David Luan,Dario Amodei,Ilya Sutskever,et al.Language models are unsupervised multitask learners.OpenAI blog,1(8):9,2019.Syama Sundar Rangapuram,Matthias W Seeger,Jan Gasthaus,Lorenzo Stella,Yuyang Wang,and Tim Januschowsk

23、i.Deep state space models for time series forecasting.Advances in neural information processing systems,31,2018.David Salinas,Valentin Flunkert,Jan Gasthaus,and Tim Januschowski.Deepar:Probabilistic forecasting with autoregressive recurrent networks.International Journal of Forecasting,36(3):1181119

24、1,2020.Ruofeng Wen,Kari Torkkola,Balakrishnan Narayanaswamy,and Dhruv Madeka.A Multi-Horizon Quantile Recurrent Forecaster.arXiv:1711.11053,2017.Chen Xu and Yao Xie.Conformal Prediction Interval for Dynamic Time-Series.In International Conference on Machine Learning,pp.1155911569.PMLR,2021.Hao Xue a

25、nd Flora D.Salim.PromptCast:A New Prompt-based Learning Paradigm for Time Series Forecasting.arXiv:2210.08964,2023.Ansari,A.F.,Stella,L.,Turkmen,C.,Zhang,X.,Mercado,P.,Shen,H.,.&Wang,Y.(2024).Chronos:Learning the language of time series.arXiv preprint arXiv:2403.07815.1/7/25www.gridaxon.io 15XFRTran

26、sformComputingEnabling Widespread Use of Local Renewable Generation for New Large Loads via Explainable AISteve Reinhardtstevexfr.aiTransform Computing,Inc.XFRTransformComputingSupporting new large loads via local renewable generation(with storage)is an ideal solution.XFR de-risks and accelerates it

27、s practical adoption via explainable-AI-based analytics for mesogrids.XFRTransformComputingXFRTransformComputingAgendaWhy is renewable generation(plus storage)ideal for new large loads?What precludes it already being deployed today?How do(scaled)microgrids benefit the grid transition?How does XFRs e

28、xplainable AI accelerate widespread use of local renewable generation for new large loads?How does XFRs explainable AI work?When is XFRs explainable AI deployed?XFRTransformComputingXFRTransformComputingLocal Renewable Generation is an Ideal Solution for New Large LoadsSolar and wind are predicted t

29、o be the cheapest energy sources even when factoring in storage required by their intermittencyMinimal climate footprintAvoids interconnection delayshttps:/ Preventing Renewable Mesogrid Deployment mesogrid=a microgrid(architecturally)scaled to large sizeRenewable generation is intermittent(i.e.,not

30、 firm)and hence requires storage,and management of generation/storage/load at scale is hardStorage is too expensive for large loads but expected to dropUntil recent inability to find power for new AI data centers,little urgency to press renewable generation this hardHence,little focus on scaling and

31、 stressing microgrids and analyticsBattery cost projections for 4-hour lithium-ion systemsCole,NREL,https:/www.nrel.gov/docs/fy23osti/85332.pdfXFRTransformComputingXFRTransformComputingArchitectural Value of Microgrids in Energy TransitionMicrogrid=a set of interconnected loads and distributed energ

32、y resources that acts as a single controllable entity with respect to the grid.It can operate in either grid-connected or in island mode.Ongoing,delivers resilience via separation/reconnectionLimits the impact of failures in a single microgridDelivers local ability to survive wider failuresDuring th

33、e energy transition,enables portions of the grid to evolve at different rates.Portions that are less well served can accept greater risk to explore new high-value capabilities.Portions that are more reliability-focused dont have to move as fast.XFRTransformComputingXFRTransformComputingXFRs Analytic

34、s Accelerate Mesogrid AdoptionKey analytic issues limiting mesogrid adoptionElectromagnetic transient(EMT)detection and mitigationSituational awareness esp.of emergent issuesCybersecurity within the mesogridBased on XFRs innovative explainable-AI(XAI)capabilitiesBasic AI is inscrutable;humans direct

35、ing high-value activities need rationale for actionXFR complements AI(language models)with advanced math(e.g.,persistence diagrams)Advanced math provides a math sanity check on the AI results Language model translates between grid concepts and(obscure)math concepts to empower grid operatorsXFRTransf

36、ormComputingXFRTransformComputingReturn to Overview slideUse Case:Enabling Grid Operator to Respond to Mesogrid Issues via XAI Human grid operator can specifydesired change in grid concepts Grid Assistant translates(via LM)to code/math and simulates,then converts baseline and modified results each t

37、o persistence diagrams and passes both to LM as evidence of the benefit;LM uses evidence to recommend Persistence diagrams readily summarize large dataXAI:a)use LM to translate between grid and math/code concepts,and b)use LM to interpret persistence diagrams XFRTransformComputingXFRTransformComputi

38、ngXFR Grid Assistant Next Steps Prototype working on small grid configuration;needs validation,scaling,etc.XFR seeking partners with urgency to deploy mesogrids to give feedback as we evolve the prototypeTarget mesogrids:renewably powered AI data centers,military bases XFR seeking angel investorsXFR

39、 seeking renewable-grid expertise via product manager and lead engineerXFRTransformComputingXFRTransformComputingBackup 2025 Electric Power Research Institute,Inc.All rights Agnieszka CzeszumskaTechnical Leader I,EPRI Europe DACAI/DX SummitDecember 7th,2025Beyond the chatbotLLM:friend of foe?2025 El

40、ectric Power Research Institute,Inc.All rights reserved.27Data-Driven Decision Making(3DM)Applying Data Science in the Nuclear Power IndustryLeverage data science for the Nuclear Power industryLaunch&support activities across the Nuclear SectorGeneral application areas Insights:learning from the pas

41、tPrognostics:anticipating the futureAutomation:increasing reliabilityOptimization:increasing efficiencyMore details on 3DM program pageProjectsResults 2025 Electric Power Research Institute,Inc.All rights reserved.28 Initiate key proof of concepts Sponsor early implementation&tech transferLaunchOver

42、all ApproachProof of ConceptEarly ValueDeployment3DMHome R&D Issue Program AI projects across the R&D sectors Strategic initiativesSupportSoftware 2025 Electric Power Research Institute,Inc.All rights reserved.29Current R&D internal PoC projects that 3DM is involved in Operating Experience(OE)Consol

43、idation search tool in testing,summarization and data formatting in progressPart 21 data ingested,data extraction and cleaning in progressAI-assisted Fuel Failure Identification work ongoingNuclear Decommissioning Knowledge Management data ingested,evaluations ongoing,UX work startedNMAC Troubleshoo

44、ting Tool data ingestion started 2025 Electric Power Research Institute,Inc.All rights reserved.30Part 21 Reports DatabasePI:Allison Read 2025 Electric Power Research Institute,Inc.All rights reserved.31Part 21 Reporting of Defects and NoncompliancePart 21 reports are submitted to the NRC.These repo

45、rts are required when a licensed facility,activity,or basic component contains defects or failures to comply with NRC regulations that could create a substantial safety hazard.1000s of reports from NRC website in pdf format are difficult to parse through and search for relevant information quicklyNa

46、tural Language Processing 2025 Electric Power Research Institute,Inc.All rights reserved.32Part 21 ReportsThe goal is to extract each section(i)-(ix)and information from them into a structured databaseMany text extraction challengesAccession Number(i)(ii)(ii)ML101820160GE Hitachi Nuclear EnergyThe b

47、asic component that failed is a High Pressure Coolant Injection SystemThe diaphragms are manufactured by Chicago-Allis as commercial grade.They are dedicated and sold as safety-related by GE Hitachi Nuclear Energy 2025 Electric Power Research Institute,Inc.All rights reserved.33Processing PDFsPDFs a

48、re processed to remove footers and headers,and text saved into a database for further processingSome PDFs are unreadable,and some spelling mistakes occur due to processing(i.e.1 instead of I)tder,tifition Jl the:btJs.fcX)mponent supplled for such,facility or such activity within the United.Stat.,:s,

49、whir;tJ may feil to comply,or co,talns pot(;)nti defect._ 2025 Electric Power Research Institute,Inc.All rights reserved.34Extraction with standard NLP techniques“Standard”NLP:Using regular expressions and fuzzy search to grab specific sectionsChallenge 1:Two-thirds of sections missingSome reports d

50、ont follow the standard structureChallenge 2:More granular information such as part number would be usefulCan LLMs be used to extract information and summarize it instead?2025 Electric Power Research Institute,Inc.All rights reserved.35Challenge 1:Missing sectionsUse semantic search to grab relevant

51、 information and extract dataEquivalent to using RAG approachAddress challenge 2 as wellRequires prompt-tuning,i.e.designing the prompt that nudges the LLM to answer correctly,in a specific format(i.e.JSON)https:/ 2025 Electric Power Research Institute,Inc.All rights reserved.36Challenge 2:Extract s

52、pecific information with LLMsAdds value to the database by classifying and grabbing specific information,for example model numbersLLM often not reliableHallucinations can be quite obvious but not always!https:/ 2025 Electric Power Research Institute,Inc.All rights reserved.37Recovering missing secti

53、onsAble to extract x2 more dataAccuracy needs to be checked against ground truth data!2025 Electric Power Research Institute,Inc.All rights reserved.38Evaluating text:metrics to compare with ground truthSAS,Cosine Similarity:normally a gold standard in NLP text comparisons,but may not be applicable

54、to comparing data with no semantic meaning like part numbers of manufacturer namesFuzzy string matching:token set ratio can compare similar words,taking care of misspellings and word orderShared words(expand acronyms,remove stopwords like inc,ltd)Choose metrics based on use case and evaluate how wel

55、l they score results 2025 Electric Power Research Institute,Inc.All rights reserved.39ConclusionsSemantic search is a great tool to find relevant portions of textExtraction of specific data with LLMs needs careful evaluation and prompt tuning to achieve acceptable results,but hallucinations still pr

56、esentCombining standard NLP techniques with LLMs,under careful evaluation,can add value to databases 2025 Electric Power Research Institute,Inc.All rights reserved.40 2025 Electric Power Research Institute,Inc.All rights TOGETHERSHAPING THE FUTURE OF ENERGYUse of Large Language Models to Improve Art

57、ificial Intelligence PredictionsAhmad Al Rashdan Ph.D.DOE LWRS ProgramPlant Modernization Pathway Lead1/7/2025 Extend the life and improve the performance of the existing fleet through modernized technologies and improved processes for plant operation and power generation.Develop modernization solut

58、ions that improve reliability and economic performance while addressing the U.S.nuclear industrys aging and obsolescence challenges.Deliver a sustainable business model that enables the U.S.nuclear industry to remain cost competitive.42Our Mission AI predictions often rely on short text inputs,posin

59、g a challenge in various applications for nuclear power plants.Examples of those applications:Classification of operator logs into startup,shutdown,or failure for availability and reliability calculations.Estimation of work activities duration for outage scheduling and optimization.Screening of cond

60、ition reports(CRs)for decisions such as condition adverse to quality(CAQ)or not CAQ(NCAQ).This limitation affects the development of classical Natural Language Processing(NLP)models capable of accurate predictions.43Problem StatementHistogram of CR Lengths Solution 1:Can we use LLMs to make predicti

61、ons?44Prompt:“Based on the information given,please elaborate.”Prompt:“Log:Reset panel BKR EQUIP ID LOCATION per CODE BKR tripped due to welding in the area.Based on the information given does this appear to be a shutdown of equipment or does it not appear to be a startup of equipment?”Prompt:“Make

62、an argument that the nuclear power plant operator log is a shutdown and another argument that it is a startup.”Potential Solutions Solution 2:Large Language Models(LLMs)excel at generating context and adding background information to text.Can we use LLMs to explain short texts to improve prediction

63、accuracy?45Using Generative AI Models in Classification of Operator Log EventsPrompt:“Please answer yes or no.Is this an equipment startup:OP LOG?”Prompt:“Please only give a number between 1 and 10 as your response.A startup score of 1 means definitely not an equipment startup.A startup score of 10

64、definitely represents an equipment startup.In this nuclear power plant operator log what is the startup score only:OP LOG?”TF-IDF+Ridge Regression (No Generative AI)F1 Score Precision/Recall Confusion Matrix 87.2%Default score of 66%93.2%/82.0%Predicted False True Actual False 47 3 True 9 41 F1 Scor

65、e Precision/Recall Confusion Matrix 71.0%Default score of 66%55.7%/98.0%Predicted False True Actual False 11 39 True 1 49 Generative AI(LLMs)46Using Generative AI Models to Enhance Input for Operator Log Events ClassificationMethod Shutdown F1 Score Startup F1 Score Failure F1 Score Mean TF-IDF+Ridg

66、e 83.2 87.2 69.0 79.8 Ratings 82.4 71.0 70.1 74.5 Vicuna Explain+TF-IDF+Ridge 88.0 83.8 70.1 80.6 Vicuna Elaborate+TF-IDF+Ridge 89.3 87.7 68.3 81.8 Prompt:“Please elaborate on this nuclear power plant operator log:LOG”Prompt:“Based on the information given in this nuclear power plant operator log do

67、es this appear to be a startup of equipment or does it not appear to be a shutdown of equipment:LOG”Prompt:“Make an argument that the nuclear power plant operator log is a shutdown and another argument that it is a startup:LOG”Explain PromptsElaborate PromptResults After Using Vicuna for Explanation

68、 and Elaboration 47Actual vs.Predicted Completion Time in DaysRegression Performance Using a Sliding WindowUsing Generative AI Models to Elaborate on Input for Estimation of Condition Resolution Time 48Threshold for Elaboration(Tokens)Number of CRs That Were Elaborated CRs below Threshold All CRs R2

69、 Before Elaboration(%)R2 After Elaboration(%)R2(%)Default 0 NA NA 51.65 100 511 32.7 34.6 51.67 150 6,768 34.3 37.0 51.97 200 19,247 40.8 41.8 52.44 250 31,626 45.38 45.26 52.59 Impact of Text Elaboration on Model Performance Using Generative AI Models to Elaborate on Input for Estimation of Conditi

70、on Resolution Time Model Performance as a Function of Token Count for 511 CRs with Fewer than 100 Tokens Each49Actual vs.Predicted Completion Time in Days After CR Size ElaborationUsing Generative AI Models to Elaborate on Input for Estimation of Condition Resolution Time Actual vs.Predicted Complet

71、ion Time in Days Before CR Size Elaboration Despite their ability to understand context,generative AI models are not necessarily more accurate than classical NLP methods for prediction.Generative AI models can provide missing context that may improve prediction,but when the text is well-composed,they could confuse the predictor and result in suboptimal outcomes.An optimal combination of generative AI and NLP-based methods can be achieved through careful examination of each case.50ConclusionsSustaining National Nuclear Assetslwrs.inl.gov51