《Talkwalker：2023年Blue Silk?GPT消費智能白皮書（英文版）（17頁）.pdf》由會員分享，可在線閱讀，更多相關《Talkwalker：2023年Blue Silk?GPT消費智能白皮書（英文版）（17頁）.pdf（17頁珍藏版）》請在三個皮匠報告上搜索。

1、Talkwalkers Blue Silk GPT for consumer intelligence:New way of generating insightsBlue Silk AI white paperThe first breakthrough in AI was in the computer vision field thanks to the appearance of CNN models 2(Convolutional Neural Network),the use of backward propagation techniques3 and most importan

2、tly,larger datasets,and training those models on GPU(Graphical Processing Units)cards.In 2012,during the ImageNet Large Scale Visual Recognition Challenge,the AlexNet 4 model,combining all the above methods,drastically reduced the top-5 error rate to 15.3%,more than 10 percent lower than the other m

3、odels in the competition.In the following years,with each improvement in the architecture such as new activation functions 5 or batch normalization 6 layers,the modern state-of-the-art CNN models exceeded human performance and recently surpassed the top-5 error rate with an astonishing 99%accuracy l

4、evel 7.This staggering quantitative improvement put the field on the map and launched the artificial intelligence boom.However,even though AI was booming,its Natural Language Processing(NLP)subfield was still waiting for its revolution,namely pre-trained language models.A language model is a probabi

5、lity distribution over a sequence of words.More specifically,it outputs a probability that represents the validity of the input sentence.Those models can be used for a large number of problems.However,so far,they performed relatively poorly and each model was trained for a specific task and could no

6、t be used for other purposes.In 2017,the Transformer architecture 8 composed of self-attention layers was introduced,and it changed radically the possibilities in the field.Introduction The history of AI often dates back to the early days of computing when machines were first used to perform simple

7、calculations.However,the field of AI began to take shape in the 1950s,with the development of the first programming languages and the first attempts to create artificial intelligence.One of the earliest and best-known examples of AI is ELIZA 1,a natural language processing program that simulated a c

8、onversation.Blue Silk AI white paperBlue Silk AI white paperCompared with previous models used in the field,namely Recurrent Neural Networks(RNNs),transformers can process the full input text at once and the attention mechanism will not only interpret each word but the context around them and the in

9、terdependencies between the words.In 2018,using this architecture,the BERT 9 model was introduced.Figure 1.Language model size through the yearsIt was trained using two self-supervised training methods:Masked Language Modeling,which consists of randomly masking words in the input text in order to as

10、k the model to predict them using the context around,and Next Sentence Prediction,which consists of predicting if a chosen next sentence was probable or not given the first sentence.Thanks to this self-supervised training method,there is no need for manually labeled data during the training phase,he

11、nce it can be trained on larger datasets and can see more diverse texts.BERT has been trained on all the text from English Wikipedia and Brown Corpus.This self-supervised method allows the model to take advantage of the transformer architecture fully and is able to produce contextual embeddings for

12、words.Using those contextual embeddings and fine-tuning the model on specific tasks allow the model to greatly outperform the previous state-of-the-art models on a significant number of natural language understanding tasks.Moreover,the quality of those contextual word embeddings opened up a plethora

13、 of opportunities in different NLP domains.As a first example,we can cite text summarization,which is the process of building a concise,cohesive,and fluent summary of a lengthier text document.Before,such summaries were built using“the most important sentences”extracted from the document.Now,with Tr

14、ansformer models,we are able to do abstractive summarization,i.e.,create new sentences from the initial content.Other examples of NLP tasks are machine translation,text categorization,and many more.For example,it was the first time a model was able to be on par with humans when translating from Chin

15、ese to English 10.However,a drawback of this approach is that while the self-supervised training does not require labeled data,there is still a need for thousands of manually labeled data for fine-tuning in order to achieve strong performance on the desired task.Blue Silk AI white paperOne thing tha

16、t deserves to be highlighted in the BERT paper is that they presented 2 models,a BASE and a LARGE model with 110M(million)and 340M parameters.While they outperformed all other models in the field,the LARGE model gives significantly better results than the BASE one(average score of 82.1 for the bigge

17、r model on the GLUE tasks compared to 79.6 for the base model and 75.1 for the previous state-of-the-art models).Those results confirmed that by increasing the model size,it is possible to improve the quality of the embedding produced by the model and obtain better results when fine-tuning it on dow

18、nstream tasks.However,the generative capability,i.e.,producing human-like text,of the pre-trained models,is limited but still showed some promise.Indeed,thanks to self-supervised training,this new era of language models exhibits an abstract ability to understand the underlying structure of natural l

19、anguage.Removing the limitation on the fine-tuning part would be desirable and it would allow us to use one model to serve several purposes without the need to manually label data and spend time training the model.Blue Silk AI white paperFigure 2.Larger models use more effectively in-context informa

20、tionBlue Silk AI white paperIn the following years,bigger and bigger transformer-based models were released,such as GPT-2 11 in 2019 with 1.5B(billion)parameters,T5 12 in 2019 with 11B parameters,GPT-3 13 with 175B parameters,Megatron-Turing NLG 14 in 2022 with 530B parameters.See Figure 1,for a vie

21、w of the general trend of model size through the years.For comparison,a typical human brain has 86B neurons.With each increase in size came improvements in the models capacity to solve NLP tasks and better grasp the context of the texts.It was the start of a new chapter,the development of large lang

22、uage models(LLMs).Those models show increased capabilities in reasoning and natural language understanding capabilities.With more parameters and bigger datasets to train on,we are able to narrow or close the gap to human-level performance in many areas of natural language research,such as machine tr

23、anslation or summarization.Those models show increased capabilities in reasoning and natural language understanding capabilities.With more parameters and bigger datasets to train on,we are able to narrow or close the gap to human-level performance in many areas of natural language research,such as m

24、achine translation or summarization.Those models are able to generate text that humans have great difficulty distinguishing from the human-written text.Next-generation AI Large language models LLMs capabilities to better understand texts and to generate high-quality text opened the door to new possi

25、bilities.Using the generating capabilities of LLMs,we are able to use LLMs to work on new tasks they were not trained on.Those new tasks are tackled in a zero or few-shot learning setup.It consists of only providing a few examples(none in the case of zero-shot)of labeled data points for a new task t

26、o the model before asking it to make predictions.In Figure 2,we show the results obtained in the GPT-3 paper 15.They study the impact of different model sizes on the accuracy using zero-shot,one-shot,and few-shot learning on a new task.The task required the model to remove random symbols from a word

27、,both with and without a natural language task prompt.A prompt is a text inserted in the text we wish to feed the model with,such that it can understand the task.For example,in the zero-shot framework,if we wish to predict the sentiment of the text“I hated this movie.”,then we can add“It was”at the

28、end of the input text and ask the model to continue the sentence.It will probably generate text like“terrible”or“awful”,and those words would have been previously labeled as negative.In the case of one/few-shot,we would also add examples in the prompt along with the description of the task if desire

29、d.In the case of translation,an input given to the model would look like this:Blue Silk AI white paperIn this new era of large language models,using the zero/few-shot learning abilities,we dont need to manually label a huge amount of data in order to fine-tune a model for a specific task.In place,on

30、e model is trained in order to be able to solve multiple tasks.These techniques are a breakthrough for NLP applications where data is scarce or difficult to annotate and label.This means that by providing a carefully chosen prompt one can toggle a large language model into very distinct inference mo

31、des,such as classification or summarization.Different prompts are possible.For example,we can use natural language by simply explaining what the model should do like we would explain it to a person.For example,in the sentiment classification task,we could also use the following prompt:Decide whether

32、 the following sentence sentiment is positive,neutral,or negative.Sentence:“I hated this movie.”Sentiment:And the model should output“negative”.Another example of instruction is the one we used for the translation task,namely“Translate English to French”.Blue Silk AI white paperTranslate English to

33、French:pea otter=loutre de merpeppermint=menthe poivreplush giraffe=girafe peluchecheese=And the model should output“fromage”.Success stories Among the recent LLM advances,we should emphasize several success stories.GPT-3,an Open-AI model with 175B parameters,as we already mentioned previously,can h

34、andle an impressive amount of tasks,such as question answering,classification,machine translation,summarization,etc.The model can be prompted with various predefined instructions or given free text to complete it.This model can classify items into categories.For example,we detail below an example of

35、 a prompt for zero-shot classification:The following is a list of companies and the categories they fall into:Apple,Facebook,FedExAppleCategory:And the model will output:TechnologyFacebookCategory:Social MediaFedExCategory:DeliveryThe same model,when given the below prompt,is able to extract informa

36、tion from an email:Blue Silk AI white paperThis method indicates a major shift in the way we are interacting with our machine learning models and shows the endless possibilities given by such models.Another method is called prompt tuning where a smaller model is trained to infer the most suitable pr

37、ompt which auto-completes the user input depending on the task and provides more detailed instruction for an LLM.This greatly simplifies the user interaction with an LLM and bolsters faster adoption by speeding up the prompt choice that will give the best-desired results.Name:And,again,the model wil

38、l output:MayaMailing Address:2111 Ash Lane,Crestview CA 92002These examples show the great capabilities of LLM and the large number of tasks that such a model can address.Another recent LLM example is Googles LaMDA(Language Model for Dialogue Applications)conversational model.This model was trained

39、on dialogues and is capable of being engaged in open-ended conversations.Compared with other modern conversational agents(also known as chatbots),LaMDA is able to talk about an apparently endless number of topics and give responses that make sense and relate clearly to the context of the conversatio

40、n.The model is so good at mimicking the way humans interact that one of Googles engineers was convinced that the model was sentient with an emerging consciousness.He started an internal investigation and even went public in order to warn people.Recently,OpenAI introduced a state-of-the-art AI langua

41、ge model called ChatGPT.Derived from GPT-3,ChatGPT is designed to tackle various natural language processing(NLP)tasks,such as text generation,conversation generation,language translation,and text classification.With its vast training corpus,ChatGPT generates text that is both contextually relevant

42、and linguistically sound,making it a valuable asset for applications where generating human-like text is crucial.Blue Silk AI white paperExtract the name and mailing address from this email:Dear Kelly,It was great to talk to you at the seminar.I thought Janes talk was quite good.Thank you for the bo

43、ok.Heres my address 2111 Ash Lane,Crestview CA 92002 Best,MayaBlue Silk AI white paperFor instance,in customer service,ChatGPT-powered chatbots can offer 24/7 assistance,and in content creation,it can be used to produce creative writing.In addition to its practical applications,ChatGPT has also been

44、 the subject of academic research.The study was conducted by a professor at the University of Pennsylvanias Wharton School and it demonstrated ChatGPTs capability by showing that it was able to pass the MBA programs final exam at the university.Finally,we need to highlight yet another collaborative

45、success of OpenAI and GitHub:Copilot,which is a tool based on the GPT-3 variant called Codex.The latter is a modified production-ready version of the GPT-3 model capable of generating solution-ready programming code,describing varying input codes,and auto-completing them.We can think of Copilot as o

46、ne of the wonderful examples where LLMs are becoming an assistant to the software engineer allowing him to write routine and tedious boilerplate code much faster while concentrating on more challenging problems at work.These success stories reflect the great capabilities of LLM and the large number

47、of tasks that such models can address.Thanks to LLMs,we can now create a set of tools centered around one single model that can produce very good results on problems that were previously hard to solve,e.g.,zero-shot classification or human-level machine translation.All in all these models would allo

48、w businesses to expand and strengthen their expertise in ambitious market exploration phases,where new emerging niches are being backed by LLMs.Applications for consumer intelligence Large language models can be used to achieve long-standing goals in consumer intelligence and social listening.This m

49、eans that LLMs offer a quick deep-dive into the massive amount of data that is generated by consumers every single day.Here we present two first examples powered by Talkwalkers Blue Silk GPT technology Blue Silk InsightA common challenge in consumer intelligence is quickly identifying key findings i

50、n huge amounts of unstructured data.Blue Silk Insight is the first technology on the market that leverages large language models to seamlessly extract key insights and succinctly write a fluent summary of thousands of reviews and comments in a matter of seconds.Blue Silk Insight tackles this monumen

51、tal task by leveraging the models ability to identify and synthesize recurrent themes into deliverable insights about what your consumers love and hate about your product,without the hassle of manually digging through conversations,word clouds and complex filters.1-Click AI ClassifiersUsing the impr

52、essive zero-shot classification capability of LLMs displayed in the previous Success stories section,our customers are able to categorize their data with minimal effort and without the need of labeling hundreds of documents.Our AI engine powered by Talkwalkers Blue Silk GPT technology is moving on t

53、o the next stage and instead of asking our clients teams to spend time annotating data,they only need to detail in natural language how they want to segment their data,e.g.,for documents speaking of Apple,they can easily be able to differentiate between the company and the fruit using a properly def

54、ined prompt.Blue Silk AI white paperOther applications involve the ad-hoc analysis of product quality by segmenting the data for example into categories specifically tailored for food like taste,flavor,texture just by describing the classes in natural language and therefore allowing instantly a very

55、 flexible and deep analysis of consumer behavior.Large language models in practical applications In practice,despite all the success stories and new opportunities brought by large language models,being able to achieve such results is not straightforward.Indeed,before one can use such a model,it need

56、s to be trained.Then,it needs to be accessible when we want to ask the model to make predictions and,finally,we need to formulate correctly our requests if we wish to fully benefit from the multi-task capabilities of the model.We detail in the next part the main pain points for all those steps.Hardw

57、are constraintsCompared to traditional deep learning models,LLMs require access to data centers capable of hosting them.Whereas previous deep learning models could fit in a single machine or server,now a single LLM must be spread across dozens,and sometimes hundreds,of servers due to its size.For ex

58、ample,GPT-3,with its 175 billion parameters,is more than 100 times larger than historically used deep learning models.With this infrastructure requirement comes a massive financial cost.In addition,deep technical expertise in how the model is designed,distributed computing and software engineering a

59、re required to properly and efficiently distribute the model across all servers.Blue Silk AI white paperUsing Talkwalkers Blue Silk,one doesnt need to allocate and coordinate all the resources mentioned above at considerable economic cost because it is fully incorporated into the environment and one

60、 can enjoy the LLM capabilities together with all other products on Talkwalker.For example,training a model like GPT-3 would easily cost around 10 or 20 million dollars for server allocation and power consumption.Challenges in training an LLMWith regard to training,as explained above,it is necessary

61、 to have access to a well-equipped data center,but also to have the knowledge and data to train the model properly in order to achieve high generalization capabilities,i.e.,the ability to perform well on new tasks that the model has not encountered during the training phase.Indeed,this deep technica

62、l expertise is necessary to pre-train,fine-tune,and calibrate all the large-scale details of the training that need to be orchestrated across all servers.The training time for such a model can take several months,so a well-trained team of deep-learning researchers,software engineers,and technicians,

63、all working together,is crucial to get the most out of the training.Again,the accumulated experience of Talkwalkers Blue Silk research team allows us to make the very best use of available resources and obtain a high-end model.When it comes to predictionLast but not least is the inference part,i.e.,

64、when we use both the multi-task and zero-shot capabilities of the trained LLM to tackle new use cases for consumer intelligence.To extract information from all the unstructured data present on social media,a detailed analysis has to be done by our product team to identify the most interesting use ca

65、ses.Then,the best prompts for such tasks have to be found because not all prompts work well.Research and optimization must be carried out in order to achieve high-quality results.Therefore,the experience and insight gained by Talkwalkers Blue Silk team over the years with such models allow us to qui

66、ckly provide a solution to our customers.Blue Silk AI white paperConclusion The emergence of transformer models and,more recently,of large language models has reshaped the field of NLP and offered new perspectives such as zero/few-shot learning and prompting which dramatically reduces the time neede

67、d to tackle new tasks.However,before taking advantage of the multi-tasking capabilities of an LLM,several challenges need to be addressed such as,for example,infrastructure constraints,model training,and a wise choice of prompt in order to obtain good results.For this reason,our extensive experience

68、 and massive investment into transformer models and LLMs allowed us to develop Talkwalkers Blue Silk GPT technology as a premium solution for faster extraction of more insightful information from the massive amount of data provided every day in the world of consumer intelligence.Blue Silk AI white p

69、aperBlue Silk AI white paperReference Page 1 Weizenbaum,Computer Power and Human Reason.2 Lecun et al.,“Gradient-Based Learning Applied to Document Recognition.”3 Rumelhart,Hinton,and Williams,“Learning Representations by Back-Propagating Errors.”4 Krizhevsky,Sutskever,and Hinton,“ImageNet Classific

70、ation with Deep Convolutional Neural Networks.”5 Hendrycks and Gimpel,“Gaussian Error Linear Units(GELUs).”6 Ioffe and Szegedy,“Batch Normalization:Accelerating Deep Network Training by Reducing Internal Covariate Shift.”7 Yuan et al.,“Florence.”8 Vaswani et al.,“Attention Is All You Need.”9 Devlin

71、et al.,“BERT.”10 Hassan et al.,“Achieving Human Parity on Automatic Chinese to English News Translation.”11 Radford et al.,“Language Models Are Unsupervised Multitask Learners.”12 Raffel et al.,“Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.”13 Brown et al.,“Langu

72、age Models Are Few-Shot Learners.”14 Smith et al.,“Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B,ALarge-Scale Generative Language Model.”15 Brown et al.,“Language Models Are Few-Shot Learners.”Free demoThe#1 Consumer Intelligence company The world is changing.Consumers are more dema

73、nding,more urgent,and more unpredictable than ever,and brands are struggling to keep up.Talkwalkers leading Consumer Intelligence Acceleration Platform helps you stay ahead by turning internal and external data into consumer insights that grow your brand.Over 2,500 global brands trust Talkwalker,and

74、 our international team of experts,to guide them in making the most of every opportunity in this fast-paced world and accelerate their brand growth.talkwalker We bring together everything you need to help you get closer to your consumers than ever before:Accelerated data coverage and scale integrations Market-leading AI capabilities Platform services that elevate data to intelligence Deep dive human and cultural insight from our team of strategists