預訓練時代的機器翻譯.pdf

1、DistributionsLanguageKupMAAMuecKuii aadaBMTA7CyrillidLatin alphabetOtherArabiOtherabjatSyllabarieMore than 5000 different languages in the world#page#Machine Translation has increasedinternational10%trade byoversmaller than 26%Equality to make the worldMANAGEMENT SCIENCEinformsVol.65，No.12,December2

2、019，pp.5449-5460htp:/pubsonline.infoorgjouralmnsoISSN0025-1909（print），ISSN1526-5501（online）Does Machine Translation Affect International Trade？ Evidencefrom a Large Digital PlatformErikBrynjofssonXiangHuMeng LiSloan School of Management Massachusetts Institute of TechnologyCambridgeMassacusetts 0214

3、2MarketngOlin SchoolofBusiness，Washington University in St.Louis，St.Louis，Missouri63130Contacterikbmitedu.http:/orcid.org/0000-0002-8031-6990（EB）:huiwusti.edu.http:/orcid.org/0000-0001-7595-3461（XHmenglwustl.edu，http:/orcid.org/0000-0002-5512-7952（ML）Abstract.Artificialinteligence（AI)issurpassinghum

4、anperformanceinagrowingnumberReceived:April18.2019Revised:April 18.2019of domains.However，there is limitedevidence of itseconomic effects.Usingdata fromaAccepted:April182019digital platformwestudy a key application ofAlmachine translation.Wefind that thePublished Online inArticles inAdvanceintroduct

5、ion ofanewmachine translation system has significantly increased internationalSeptember3.2019effectsare consistent withasubstantialreductionin translation costs.Ourresults providehtps:fdoi.org/10.287/mnsc.2019.3388Copyright:2019INFORMSbegun to improve economic efficiency inat least one domain.Histor

6、y:Accepted by JoshuaGans，businessstrategy.Supplemental MateriaTheonlineappendix isavailableathttps:/doi.org/10.1287/mnsc.2019.3388.Keywords:artificial intelligenceinternatiional trademachinetrranslationmachinne learningdigital platforms#page#Global Footprints of BytedanceSupportingServe in75+150+lan

7、guagescountries13230+R&DOfficescentersworldwideGlobaldTBabeTikTokVigo VideoHeloTopBuzZbuzzvideoNews RepubicChina頭條ToutiaoDouyinHuoshanFaceuTuChongXiigua VideoDon Che Di#page#Machine Translation: Conditional SequenceGenerationDecoderEncoderHowareyou1BeamSearchl1DecodarDecodaDacodarCEncodBLayorLayorLa

8、y1一1口1DococerDooodorDecodarLayer1Laye11！DecocarDecoderDacodar1一1BOSSMaximize the conditional generation probability:mpe（YIx)=IIp(:lDeModslModelsEmZh21#page#Multilingual MT Jointly？Why Training Data scarcity for low/zero resource languagesData distribution over lanJina.kiamanidagheil vecer1000000000a

9、z0pagisodobarggoodbonjourSaivaa五865uu100000000menyegabaradobeomoiennamaskaranQhallo00anmbondobry1000000021000000100000High Resource LoW Resource10000French，German，Spanish，jYoruba，Sindhi，Hawaian，22Arivazhagan etal.2019#page#Multilingual MT Jointly？Why TrainingData scarcity for low/zero resource langu

10、ages.Transfer knowledge between languages.year11 year3 months1 year23#page#What dowant？weWe want a universal pre-trained model forNMT.across all language pairs.A24#page#Why is thisinteresting2We want a universal pre-trainedMTmodelfor many languages， which adaptseasily todownstream tasks.NEREnDeFinet

11、uneFinetumeQAZhJpPMEPLNLINHPtBERT25#page#FurtherPursuit:Unified MultilingualRepresentationFurther: It is expected to bridge distributionalrepresentation of different languages.Utterances in different languages with the samesemantics will be mapped to adjacent embeddingspaces.ESEnDe llove you.Jetaime

12、寶 lch liebe dich. Te quiero.tiamo.26#page#MethodsExistingPre-trainingBERT/GPT:Partially pre-trainingGPTBERTEncederDecoderEncederDecoderJDevlin etal.2018ARadfordet al.201827#page#ExistingMethodsPre-trainingWe want a BERT-like pre-training model forNMTMASS/BART: Training objective discrepancyexists !！

13、BARTMASSX1X2X3X4X2X3X4X5EncoderEncoderDeeoderDecoderX1X3X4X3X228#page#ApproachmRASPmRASP: multilingual Random AlignedSubstitution Pre-training- Multilingual Pre-training Approach- RAS: specially designed training methodto align semantic embeddingsEncederDeooderY3XX329#page#IntuitionPre-training ines

14、sence gets anaverage model forall language pairs.OEn-FrFine-tuning further6avcderives specializedmodelsObe-EnORo-ESObe-Fr0n-De30#page#ArchitectureTransformer-big: 6 encoder， 6 decoderEncoderDecoderHowareyouBeamLinoaLinaa1Searchl1IDecoderDecoderDecoderEncJaAB7LayerJa87Jake7LayarI1DecoderDacodaDacodar

15、EncoderIMu-HeadDecoderDecoderDecodeLayeMuli-HeaoAttentionBOSMaskedMulti-Hea31mRASP is a traiiiing approach#page#Overview of mRASPJadoredanserchanteretPre-training555JadoredanserchanterEncoderDecoderOrigENidsFRidJadorechanterdanserjikesinginganddancingettok3305O24posRASHEE388888H8888T8888888888888688

16、H3likealope.rchanteretdanserchanteranddansertok3023025pos2#page#Overview of mRASPdanserJadorechanteretPre-training555JadoreetdanserchanterEncoderDecoderOrigEN idsFRidsJadorechanterdanserikesinginganddancingetok8305244pos中RASEH8886T8588088888888H888Jadorechanteretdanserchanteranddansertok23023504posL

17、anguage indicator33#page#Overview of mRASPJadoredanserchanteretPre-training555JadoredanserchanterEncoderDecoder6H0ENidsFRidJadorechanterdansersingingdancindetok2303O4posRASDOOe8888808888888688HBQ883ENidsJadorechanteretdanserdanserchartetok02304pos3RandomAligned Substitutionllove you.JetaimeEEsEnlch

18、liebe dich.34Te quieroyamo.De#page#Overview of mRASPJadoredanserchanterPre-training555JadoreetdanserchanterEncoderDecoderOrigENidsFR idsJadorechanterdanserjikesinginganddancingettok3230504posRASHEE388888HOOD0080H8888888688H80ENidslikeJadorechanteretdanserchanteranddansertok30230254posEn-FrFine-tunin

19、gEOSJadorejouerbasketballauEncoderDecoderENidslikebasketballFR idplayingJadorejouerbasketbaltokaU232300pos1#page#MethodRAS TrainingRandom AlignedSubstitution （RAS）- Randomly replace a source word to itssynonym in different language.一 Draw the embedding space closer.likeandsingingdancing蘇G#page#Metho

20、dRAS TrainingRandom AlignedSubstitution （RAS）- Randomly replace a source word to itssynonym in different language.一 Draw the embedding space closer.likesinginganddancingchanterdanser37#page#MethodRAS TrainingRandom AlignedSubstitution （RAS）- Randomly replace a source word to itssynonym in different

21、language.一 Draw the embedding space closer.likesinginganddancingchanterdanser88#page#MethodRAS TrainingRandom AlignedSubstitution （RAS）- Randomly replace a source word to itssynonym in different language.一 Draw the embedding space closer.likeandchanterdansersingingdancingCpre=x)Dll-logre(x1C(x）i.je8

22、39#page#Top 3 QuestionsDoes mRASPwork for bothlow-resourceand high-resource scenarios？ Does mRASP work when encounteringunseen languages7why mRASP works？40#page#DatasetsPre-training Dataset: PC32 (Parallel Corpus 32）-32 English-centric language pairs， resulting in64 directed translation pairs in tot

23、al-Contains a total size of 110.4M public parallesentence pairs1sze1000000001000000Q1000000100000100001000100101Fr 7h Iv De FiViBot41#page#DatasetsFine-tuning Dataset Indigenous Corpus: included in pre-training phase-Extremely low resource （100k and 1M）(He，Tr， etc.）-Medium resource (1M and Fr +1.1BL

24、EU）.4731口DirectL CTNMTL CTNMT口Direct口mBART口 mRASP口XLM口 MASS口 mRASP口mBERT45.2530.2543.529.541.7528.75402846En2Fr（wmt2014）En2De（wmt2016）#page#performance forDoes mRASP boost MTExotic Languages2 mRASP generalizes on all exotic scenarios.Fr-Zh(20K）De-Fr(9M）一1v321.20.723.5DirectExotic Pair23.425.826.729.

25、9mRASPDa-EI(1.2M）NI-Pt(12K）Ki1V1V0.016.9Direct0.014.1Exotic Full19.9mRASP14.113.217.6En-Mr(11K）En-Gl(1.2M）一）丨1v1VDirect6.46.88.912.822.922.738.1mRASP32.1ExoticEn-Eu（726k）En-SI(2M）Source/Target1V1V7.1Direct10.924.228.24728.427.629.5mRASP199.1#page#performance forDoes mRASP boost MTExotic Languages2 m

26、RASP generalizes on all exotic scenarios.Fr-Zh(20K）De-Fr(9M）一1v321.20.723.5DirectExotic Pair23.429.9mRASP25.826-ZDa-EI(1.2M）NI-Pt(12K）KiI1）1）16.9Direct0.00.014.1Exotic Full19.9mRASP17.614113.2.En-M（11K）En-Gl(1.2M）丨一）1V1VDirect6.46.88.912.822.922.738.1mRASP32.1ExoticEn-Eu726k）En-SI(2M）iFr）:+1.1 BLEU

27、gains，-Exotic FullNI-Pt,12K）:010+ BLEU gains.mRASP takes a step towards the universalsemantic representation.- There is certain connection between the semanticrepresentation being bridged and MT models being improved.6#page#Thanks!mRASPCO Code and models available at:- https:/ thank Liwei Wu， Huadong ChenQiangian Dong， Zewei Sun， Yang Wei andWeiying Ma for their useful suggestions57