3 jf@sdZddlmZddlmZddlmZddlmZddlmZmZm Z m Z m Z ddl Z ddl mZmZdd lmZmZdd lmZdd lmZdd lmZed ZeedBZedZeeBZeedZeedZeedBedZeeBZ eedBZ!e!eBZ"e"edZ#ddZ$Gddde%Z&Gddde Z'Gddde'Z(Gddde'Z)Gdd d e'Z*Gd!d"d"e'Z+Gd#d$d$e(Z,Gd%d&d&e'Z-Gd'd(d(e'Z.Gd)d*d*e'Z/Gd+d,d,e'Z0Gd-d.d.e0Z1Gd/d0d0e(Z2Gd1d2d2e'Z3Gd3d4d4e'Z4Gd5d6d6e'Z5Gd7d8d8e'Z6Gd9d:d:e'Z7Gd;d<dd>e'Z9Gd?d@d@e'Z:GdAdBdBe'Z;GdCdDdDe'ZGdIdJdJe'Z?GdKdLdLe'Z@GdMdNdNe'ZAGdOdPdPe*ZBGdQdRdRe'ZCGdSdTdTe'ZDGdUdVdVe'ZEGdWdXdXe'ZFGdYdZdZeFZGGd[d\d\e'ZHGd]d^d^e'ZIGd_d`d`e'ZJGdadbdbe'ZKGdcdddde'ZLGdedfdfeLZMGdgdhdheLZNGdidjdje'ZOGdkdldle'ZPGdmdndne'ZQGdodpdpe ZRGdqdrdreRZSGdsdtdteRZTGdudvdveSZUeTddwZVeTdxdyZWeTdzd{ZXe jYd|jZd}j[ej\Z]e jYd~jZd}j[ej^ddj^ddj_Z`e jYdjaZbe jYd~jZd}j[e j^ddj^ddj_Zce jYd~jZd}j[e"j^ddj^ddj_Zde jYd~jZd}j[e#j^ddj^ddj_ZeddZfddZgddZhddZiddZjddZkddZlddZmddZnddZoddZpddZqddZrddZsddZtddZuddZvddZwddZxddZyddZzddZ{ddZ|ddZ}ddZ~ddZddZddZddZddZddZddÄZddńZddDŽZddɄZdd˄Zdd̈́ZddτZddфZddӄZddՄZddׄZddلZddۄZdd݄Zdd߄ZddZddZddZddZddZddZdS)alHeader value parser implementing various email-related RFC parsing rules. The parsing methods defined in this module implement various email related parsing rules. Principal among them is RFC 5322, which is the followon to RFC 2822 and primarily a clarification of the former. It also implements RFC 2047 encoded word decoding. RFC 5322 goes to considerable trouble to maintain backward compatibility with RFC 822 in the parse phase, while cleaning up the structure on the generation phase. This parser supports correct RFC 5322 generation by tagging white space as folding white space only when folding is allowed in the non-obsolete rule sets. Actually, the parser is even more generous when accepting input than RFC 5322 mandates, following the spirit of Postel's Law, which RFC 5322 encourages. Where possible deviations from the standard are annotated on the 'defects' attribute of tokens that deviate. The general structure of the parser follows RFC 5322, and uses its terminology where there is a direct correspondence. Where the implementation requires a somewhat different structure than that used by the formal grammar, new terms that mimic the closest existing terms are used. Thus, it really helps to have a copy of RFC 5322 handy when studying this code. Input to the parser is a string that has already been unfolded according to RFC 5322 rules. According to the RFC this unfolding is the very first step, and this parser leaves the unfolding step to a higher level message parser, which will have already detected the line breaks that need unfolding while determining the beginning and end of each header. The output of the parser is a TokenList object, which is a list subclass. A TokenList is a recursive data structure. The terminal nodes of the structure are Terminal objects, which are subclasses of str. These do not correspond directly to terminal objects in the formal grammar, but are instead more practical higher level combinations of true terminals. All TokenList and Terminal objects have a 'value' attribute, which produces the semantically meaningful value of that part of the parse subtree. The value of all whitespace tokens (no matter how many sub-tokens they may contain) is a single space, as per the RFC rules. This includes 'CFWS', which is herein included in the general class of whitespace tokens. There is one exception to the rule that whitespace tokens are collapsed into single spaces in values: in the value of a 'bare-quoted-string' (a quoted-string with no leading or trailing whitespace), any whitespace that appeared between the quotation marks is preserved in the returned value. Note that in all Terminal strings quoted pairs are turned into their unquoted values. All TokenList and Terminal objects also have a string value, which attempts to be a "canonical" representation of the RFC-compliant form of the substring that produced the parsed subtree, including minimal use of quoted pair quoting. Whitespace runs are not collapsed. Comment tokens also have a 'content' attribute providing the string found between the parens (including any nested comments) with whitespace preserved. All TokenList and Terminal objects have a 'defects' attribute which is a possibly empty list all of the defects found while creating the token. Defects may appear on any token in the tree, and a composite list of all defects in the subtree is available through the 'all_defects' attribute of any node. (For Terminal notes x.defects == x.all_defects.) Each object in a parse tree is called a 'token', and each has a 'token_type' attribute that gives the name from the RFC 5322 grammar that it represents. Not all RFC 5322 nodes are produced, and there is one non-RFC 5322 node that may be produced: 'ptext'. A 'ptext' is a string of printable ascii characters. It is returned in place of lists of (ctext/quoted-pair) and (qtext/quoted-pair). XXX: provide complete list of token types. )print_function)unicode_literals)division)absolute_import)intrangestrsuperlistN) namedtuple OrderedDict)unquoteunquote_to_bytes)_encoded_words)errors)utilsz (z ()<>@,:;.\"[].z."(z/?=z*'%%cCs dt|jddjdddS)N"\z\\z\")rreplace)valuerM/tmp/pip-build-s1i5078x/future/future/backports/email/_header_value_parser.py quote_stringdsrc@s>eZdZddZddZddZddZd d Zdd d Zd S)_FoldedcCs0||_||_d|_d|_d|_g|_t|_dS)NrT)maxlenpolicylastlen stickyspace firstlinedoner current)selfrrrrr__init__msz_Folded.__init__cCs2|jj|j|jj|jj|jjd|_dS)Nr)r"extendr#appendrlinesepclearr)r$rrrnewlinevs z_Folded.newlinecCs|jr|jdS)N)r#r*)r$rrrfinalize|sz_Folded.finalizecCs dj|jS)N)joinr")r$rrr__str__sz_Folded.__str__cCs|jj|dS)N)r#r')r$stokenrrrr'sz_Folded.appendNcCs|dkrt|}t|}|jdk rt|j}|j|||jkr|jj|j|j|7_|jj||j|7_d|_d|_dS|jr|j }|dk r|jt|7_|t|7}|j |dS|o|d|jkrp|j|}d|ko|knr4||}|jj|jd||j|d|_|}|j |jj|j|jj||||_d|_d|_dS|js|j |jj|j|jj|d|_d|_dS|j||jkr|jj||j|7_dS||jkr|j |jj|||_dSdS)NFTr) rlenr rrr#r'r!has_fwspop_leading_fws_foldr*)r$tokenr/lZstickyspace_lenwsmarginZtrimrrrappend_if_fitssf            z_Folded.append_if_fits)N) __name__ __module__ __qualname__r%r*r+r.r'r9rrrrrks  rcseZdZdZfddZddZfddZedd Zed d Z ed d Z ddZ ddZ ddZ eddZddZeddZddZddZddZd d!Zd)d#d$Zd*d%d&Zd+d'd(ZZS), TokenListNcstt|j||g|_dS)N)r r=r%defects)r$argskw) __class__rrr%szTokenList.__init__cCsdjdd|DS)Nr,css|]}t|VqdS)N)r).0xrrr sz$TokenList.__str__..)r-)r$rrrr.szTokenList.__str__csdj|jjtt|jS)Nz{}({}))formatrAr:r r=__repr__)r$)rArrrFs zTokenList.__repr__cCsdjdd|DS)Nr,css|]}|jr|jVqdS)N)r)rBrCrrrrDsz"TokenList.value..)r-)r$rrrrszTokenList.valuecCstdd|D|jS)Ncss|] }|jVqdS)N) all_defects)rBrCrrrrDsz(TokenList.all_defects..)sumr>)r$rrrrGszTokenList.all_defectsccs|j}t}xd|D]\}|jrH|rHt|dkr6|dn||V|j|j}|j||r||V|g}qW|rt|dkr|dn||VdS)Nr0r)rAr startswith_fwsr1r)pop_trailing_wsr')r$klassthisr5Zend_wsrrrpartss    zTokenList.partscCs |djS)Nr)rI)r$rrrrIszTokenList.startswith_fwscCs$|djdkr|jdS|djS)Nrfws) token_typepopr3)r$rrrr3s zTokenList.pop_leading_fwscCs$|djdkr|jdS|djS)Nr0cfwsrRrR)rOrPrJ)r$rrrrJs zTokenList.pop_trailing_wscCsx|D]}|jrdSqWdS)NTF)r2)r$partrrrr2s zTokenList.has_fwscCs |djS)Nr)has_leading_comment)r$rrrrT"szTokenList.has_leading_commentcCs"g}x|D]}|j|jq W|S)N)r&comments)r$rUr5rrrrU%s zTokenList.commentscKs@|d}|d=|jptd}t||}|j||jt|S)Nrz+inf)max_line_lengthfloatrr4r+r)r$Z _3to2kwargsrrfoldedrrrfold,s  zTokenList.foldcCs`g}|j}|r|j||djdkr2|jdnd}|jtjt|||j|dj|S)Nr0rNr,rRrR)r3r'rOrP_ewencoderr-)r$charsetresr7trailerrrras_encoded_word5s  zTokenList.as_encoded_wordcCs.g}x|D]}|j|j||q Wdj|S)Nr,)r' cte_encoder-)r$r\rr]rSrrrr`As zTokenList.cte_encodec Csx|jD]}t|}t|}yt|jdWnHtk rvtdd|jDrXd}nd}|j||j}t|}YnX|j ||rq|j }|dk rt|j d|_ |j |rq|j r|j|q|j||jqWdS)Nzus-asciicss|]}t|tjVqdS)N) isinstancerUndecodableBytesDefect)rBrCrrrrDNsz"TokenList._fold..z unknown-8bitzutf-8r)rMrr1r[UnicodeEncodeErroranyrGr`rr9r3rPr r2r4r'r*)r$rXrStstrtlenr\r7rrrr4Gs0      zTokenList._foldr,cCstdj|jdddS)N r,)indent)printr-_pp)r$rhrrrpprintiszTokenList.pprintcCsdj|jddS)Nrgr,)rh)r-rj)r$rhrrrppstrlszTokenList.ppstrccsdj||jj|jVxF|D]>}t|ds<|dj|Vqx|j|dD] }|VqLWqW|jrrdj|j}nd}dj||VdS)Nz{}{}/{}(rjz* !! invalid element in token list: {!r}z z Defects: {}r,z{}){})rErAr:rOhasattrrjr>)r$rhr5lineextrarrrrjos    z TokenList._pp)r,)r,)r,)r:r;r<rOr%r.rFpropertyrrGrMrIr3rJr2rTrUrYr_r`r4rkrlrj __classcell__rr)rArr=s(    +     "  r=c@s$eZdZeddZeddZdS)WhiteSpaceTokenListcCsdS)N r)r$rrrrszWhiteSpaceTokenList.valuecCsdd|DS)NcSsg|]}|jdkr|jqS)comment)rOcontent)rBrCrrr sz0WhiteSpaceTokenList.comments..r)r$rrrrUszWhiteSpaceTokenList.commentsN)r:r;r<rprrUrrrrrrs rrc@s eZdZdZddZddZdS)UnstructuredTokenList unstructuredc Cstdd|Dr|j|Sd}x|jD]}t|}d}yt|jdWntk rtdd|jDrzd}nd}|dk r tdj|j |d|gj |}t d d|j d|D}t|} t | } || |j kr |j |d=|j| || |_w*|j |}d }YnX|j||r@|r*t |j d }q*|sL|rX|j|q*|j} | dk rt| |_|j|rq*|jr|j|q*|j||jd}q*WdS) Ncss|]}|jdkVqdS)z encoded-wordN)rO)rBrCrrrrDsz.UnstructuredTokenList._fold..Fzus-asciicss|]}t|tjVqdS)N)rarrb)rBrCrrrrDsz unknown-8bitzutf-8r,css|]}t|VqdS)N)r1)rBrCrrrrDsTr0)rdZ _fold_encodedrMrr[rcrGget_unstructuredr-r#r_rHr1rr'rr9Z _fold_as_ewr3r r2rYr*) r$rXlast_ewrSreis_ewr\chunkZ oldlastlenschunklchunkr7rrrr4sX   "            zUnstructuredTokenList._foldc Csg}d}x|D]}t|}y|jd|j|Wqtk r|dkrf|j|j||t|}n*tdj||d|g}|j|jYqXqWdj|S)Nzus-asciir,) rr[r'rcr`r1ryr-r_)r$r\rr]rzrSsparttlrrrr`s   z UnstructuredTokenList.cte_encodeN)r:r;r<rOr4r`rrrrrws8rwc@s eZdZdZddZddZdS)Phrasephrasec Csd}x|jD]}t|}t|}d}yt|jdWn(tk rdtdd|jDrhd}nd}|dk o||j rJ|djdkr|j r|j d}nd }x2t |D]&\} } | jd krt | dd|| <qWt d j|j|d|gj|} t| } t| } || |jkrJ|j|d=|j| td d|jD|_w|j|}t|}d }YnX|j||r|r|j rt|jd}q|j s|jd krd}q|j|qWdS)NFzus-asciicss|]}t|tjVqdS)N)rarrb)rBrCrrrrDszPhrase._fold..z unknown-8bitzutf-8r0rQr,zbare-quoted-stringcss|]}t|VqdS)N)r1)rBrCrrrrD sTz quoted-stringrRrR)rMrr1r[rcrdrGrTrOrUrP enumeraterwryr-r#r_rr'rHrr9r4)r$rXrzrSrerfZhas_ewr\ remainderir5r|r}r~rrrr4sL    "    z Phrase._foldc Cs<g}d}d}x"|D]}t|}y|jd|j|Wntk rd}|dkrz|jsft|}|j|j||n|js |d jdkr|jr|j d }nd}x2t |D]&\} } | jdkrt | dd|| <qWt dj ||d|g} | j|g||d<YnX|js*| r|jdkrd}qWdj |S) NFzus-asciiTr0rQr,zbare-quoted-stringz quoted-stringrRrR)rr[r'rcrUr1r`rTrOrPrrwryr-r_) r$r\rr]rzr{rSrrrr5rrrrr`s4    zPhrase.cte_encodeN)r:r;r<rOr4r`rrrrrs>rc@seZdZdZdS)WordwordN)r:r;r<rOrrrrr:src@seZdZdZddZdS)CFWSListrQcCs t|jS)N)boolrU)r$rrrrTCszCFWSList.has_leading_commentN)r:r;r<rOrTrrrrr?src@seZdZdZdS)AtomatomN)r:r;r<rOrrrrrGsrc@seZdZdZdS)Tokenr5N)r:r;r<rOrrrrrLsrc@s(eZdZdZdZdZdZeddZdS) EncodedWordz encoded-wordNcCs&|jdk r|jStjt||jdS)N)cterZr[rr\)r$rrrencodedXs zEncodedWord.encoded) r:r;r<rOrr\langrprrrrrrQs rc@s4eZdZdZeddZeddZeddZdS) QuotedStringz quoted-stringcCs"x|D]}|jdkr|jSqWdS)Nzbare-quoted-string)rOr)r$rCrrrruds  zQuotedString.contentcCsBg}x2|D]*}|jdkr(|jt|q |j|jq Wdj|S)Nzbare-quoted-stringr,)rOr'rrr-)r$r]rCrrr quoted_valuejs   zQuotedString.quoted_valuecCs"x|D]}|jdkr|jSqWdS)Nzbare-quoted-string)rOr)r$r5rrrstripped_valuets  zQuotedString.stripped_valueN)r:r;r<rOrprurrrrrrr`s  rc@s$eZdZdZddZeddZdS)BareQuotedStringzbare-quoted-stringcCstdjdd|DS)Nr,css|]}t|VqdS)N)r)rBrCrrrrDsz+BareQuotedString.__str__..)rr-)r$rrrr.szBareQuotedString.__str__cCsdjdd|DS)Nr,css|]}t|VqdS)N)r)rBrCrrrrDsz)BareQuotedString.value..)r-)r$rrrrszBareQuotedString.valueN)r:r;r<rOr.rprrrrrr{src@s8eZdZdZddZddZeddZedd Zd S) Commentrtcs(djtdgfddDdgggS)Nr,rcsg|]}j|qSr)quote)rBrC)r$rrrvsz#Comment.__str__..))r-rH)r$r)r$rr.s zComment.__str__cCs2|jdkrt|St|jddjddjddS)Nrtrz\\rz\(rz\))rOrr)r$rrrrrs  z Comment.quotecCsdjdd|DS)Nr,css|]}t|VqdS)N)r)rBrCrrrrDsz"Comment.content..)r-)r$rrrruszComment.contentcCs|jgS)N)ru)r$rrrrUszComment.commentsN) r:r;r<rOr.rrprurUrrrrrs  rc@s4eZdZdZeddZeddZeddZdS) AddressListz address-listcCsdd|DS)NcSsg|]}|jdkr|qS)address)rO)rBrCrrrrvsz)AddressList.addresses..r)r$rrr addressesszAddressList.addressescCstdd|DgS)Ncss|]}|jdkr|jVqdS)rN)rO mailboxes)rBrCrrrrDsz(AddressList.mailboxes..)rH)r$rrrrszAddressList.mailboxescCstdd|DgS)Ncss|]}|jdkr|jVqdS)rN)rO all_mailboxes)rBrCrrrrDsz,AddressList.all_mailboxes..)rH)r$rrrrszAddressList.all_mailboxesN)r:r;r<rOrprrrrrrrrs  rc@s4eZdZdZeddZeddZeddZdS) AddressrcCs|djdkr|djSdS)Nrgroup)rO display_name)r$rrrrszAddress.display_namecCs4|djdkr|dgS|djdkr*gS|djS)Nrmailboxzinvalid-mailbox)rOr)r$rrrrs  zAddress.mailboxescCs:|djdkr|dgS|djdkr0|dgS|djS)Nrrzinvalid-mailbox)rOr)r$rrrrs   zAddress.all_mailboxesN)r:r;r<rOrprrrrrrrrs  rc@s(eZdZdZeddZeddZdS) MailboxListz mailbox-listcCsdd|DS)NcSsg|]}|jdkr|qS)r)rO)rBrCrrrrvsz)MailboxList.mailboxes..r)r$rrrrszMailboxList.mailboxescCsdd|DS)NcSsg|]}|jdkr|qS)rinvalid-mailbox)rr)rO)rBrCrrrrvsz-MailboxList.all_mailboxes..r)r$rrrrszMailboxList.all_mailboxesN)r:r;r<rOrprrrrrrrs rc@s(eZdZdZeddZeddZdS) GroupListz group-listcCs"| s|djdkrgS|djS)Nrz mailbox-list)rOr)r$rrrrszGroupList.mailboxescCs"| s|djdkrgS|djS)Nrz mailbox-list)rOr)r$rrrrszGroupList.all_mailboxesN)r:r;r<rOrprrrrrrrs rc@s4eZdZdZeddZeddZeddZdS) GrouprcCs|djdkrgS|djS)Nz group-list)rOr)r$rrrrszGroup.mailboxescCs|djdkrgS|djS)Nrz group-list)rOr)r$rrrrszGroup.all_mailboxescCs |djS)Nr)r)r$rrrrszGroup.display_nameN)r:r;r<rOrprrrrrrrrs  rc@sLeZdZdZeddZeddZeddZedd Zed d Z d S) NameAddrz name-addrcCst|dkrdS|djS)Nr0r)r1r)r$rrrrs zNameAddr.display_namecCs |djS)Nr0rR) local_part)r$rrrr szNameAddr.local_partcCs |djS)Nr0rR)domain)r$rrrrszNameAddr.domaincCs |djS)Nr0rR)route)r$rrrrszNameAddr.routecCs |djS)Nr0rR) addr_spec)r$rrrrszNameAddr.addr_specN) r:r;r<rOrprrrrrrrrrrs     rc@s@eZdZdZeddZeddZeddZedd Zd S) AngleAddrz angle-addrcCs"x|D]}|jdkr|jSqWdS)Nz addr-spec)rOr)r$rCrrrr s  zAngleAddr.local_partcCs"x|D]}|jdkr|jSqWdS)Nz addr-spec)rOr)r$rCrrrr&s  zAngleAddr.domaincCs"x|D]}|jdkr|jSqWdS)Nz obs-route)rOdomains)r$rCrrrr,s  zAngleAddr.routecCs&x |D]}|jdkr|jSqWdSdS)Nz addr-specz<>)rOr)r$rCrrrr2s   zAngleAddr.addr_specN) r:r;r<rOrprrrrrrrrrs    rc@seZdZdZeddZdS)ObsRoutez obs-routecCsdd|DS)NcSsg|]}|jdkr|jqS)r)rOr)rBrCrrrrvAsz$ObsRoute.domains..r)r$rrrr?szObsRoute.domainsN)r:r;r<rOrprrrrrr;src@sLeZdZdZeddZeddZeddZedd Zed d Z d S) MailboxrcCs|djdkr|djSdS)Nrz name-addr)rOr)r$rrrrHszMailbox.display_namecCs |djS)Nr)r)r$rrrrMszMailbox.local_partcCs |djS)Nr)r)r$rrrrQszMailbox.domaincCs|djdkr|djSdS)Nrz name-addr)rOr)r$rrrrUsz Mailbox.routecCs |djS)Nr)r)r$rrrrZszMailbox.addr_specN) r:r;r<rOrprrrrrrrrrrDs     rc@s,eZdZdZeddZeZZZZ dS)InvalidMailboxzinvalid-mailboxcCsdS)Nr)r$rrrrcszInvalidMailbox.display_nameN) r:r;r<rOrprrrrrrrrrr_s rcs$eZdZdZefddZZS)Domainrcsdjtt|jjS)Nr,)r-r rrsplit)r$)rArrrnsz Domain.domain)r:r;r<rOrprrqrr)rArrjsrc@seZdZdZdS)DotAtomzdot-atomN)r:r;r<rOrrrrrssrc@seZdZdZdS) DotAtomTextz dot-atom-textN)r:r;r<rOrrrrrxsrc@s@eZdZdZeddZeddZeddZedd Zd S) AddrSpecz addr-speccCs |djS)Nr)r)r$rrrrszAddrSpec.local_partcCst|dkrdS|djS)Nr0rR)r1r)r$rrrrs zAddrSpec.domaincCs<t|dkr|djS|djj|dj|djjS)Nrrr0r)r1rrstriplstrip)r$rrrrs  zAddrSpec.valuecCsLt|j}t|t|tkr*t|j}n|j}|jdk rH|d|jS|S)N@)setrr1 DOT_ATOM_ENDSrr)r$Znamesetlprrrrs   zAddrSpec.addr_specN) r:r;r<rOrprrrrrrrrr}s    rc@seZdZdZdS) ObsLocalPartzobs-local-partN)r:r;r<rOrrrrrsrcs0eZdZdZeddZefddZZS) DisplayNamez display-namecCst|}|djdkr"|jdn*|ddjdkrLt|ddd|d<|djdkrd|jn*|ddjdkrt|ddd|d <|jS) NrrQr0rRrRrRrRrRrR)r=rOrPr)r$r]rrrrs  zDisplayName.display_namecsd}|jrd}nx|D]}|jdkrd}qW|rd}}|djdksX|ddjdkr\d}|d jdks||d d jdkrd}|t|j|Stt|jSdS) NFTz quoted-stringr,rrQrsr0rRrRrR)r>rOrrr rr)r$rrCprepost)rArrrs    zDisplayName.value)r:r;r<rOrprrrqrr)rArrs rc@s(eZdZdZeddZeddZdS) LocalPartz local-partcCs&|djdkr|djS|djSdS)Nrz quoted-string)rOrr)r$rrrrs zLocalPart.valuecCstg}t}d}x|dtgD]}|jdkr.q|r^|jdkr^|djdkr^t|dd|d<t|t}|r|jdkr|djdkr|jt|ddn |j||d }|}qWt|dd }|jS) NFrrQdotr0rRrRrRrRrR)DOTrOr=rar'r)r$r]lastZ last_is_tltokZis_tlrrrrs$   zLocalPart.local_partN)r:r;r<rOrprrrrrrrs rcs0eZdZdZefddZeddZZS) DomainLiteralzdomain-literalcsdjtt|jjS)Nr,)r-r rrr)r$)rArrrszDomainLiteral.domaincCs"x|D]}|jdkr|jSqWdS)Nptext)rOr)r$rCrrrips  zDomainLiteral.ip)r:r;r<rOrprrrqrr)rArrsrc@seZdZdZdZdZdS) MIMEVersionz mime-versionN)r:r;r<rOmajorminorrrrrrsrc@s4eZdZdZdZdZdZeddZeddZ dS) Parameter parameterFzus-asciicCs|jr|djSdS)Nr0r) sectionednumber)r$rrrsection_number szParameter.section_numbercCsbx\|D]T}|jdkr|jS|jdkrx4|D],}|jdkr*x|D]}|jdkr>|jSq>Wq*WqWdS)Nrz quoted-stringzbare-quoted-stringr,)rOr)r$r5rrr param_values       zParameter.param_valueN) r:r;r<rOrextendedr\rprrrrrrrs  rc@seZdZdZdS)InvalidParameterzinvalid-parameterN)r:r;r<rOrrrrrsrc@seZdZdZeddZdS) Attribute attributecCs$x|D]}|jjdr|jSqWdS)Nattrtext)rOendswithr)r$r5rrrr's  zAttribute.stripped_valueN)r:r;r<rOrprrrrrr#src@seZdZdZdZdS)SectionsectionN)r:r;r<rOrrrrrr-src@seZdZdZeddZdS)ValuercCs2|d}|jdkr|d}|jjdr,|jS|jS)NrrQr0 quoted-stringrextended-attribute)rrr)rOrrr)r$r5rrrr7s zValue.stripped_valueN)r:r;r<rOrprrrrrr3src@s$eZdZdZeddZddZdS)MimeParameterszmime-parametersc cs|t}x\|D]T}|jjdsq |djdkr.q |djj}||krLg||<||j|j|fq Wx|jD]\}}t|}g}|ddj }xt |D]\}\}} ||kr| j jt j d| j} | jrRy t| } Wn"tk rt| dd} YnRXy| j|d} Wn"tk r4| jd d} YnXtj| rR| j jt j|j| qWd j|} || fVqpWdS) Nrrrr0z*inconsistent multipart parameter numberingzlatin-1)encodingsurrogateescapezus-asciir,)r rOrrstripr'ritemssortedr\rr>rInvalidHeaderDefectrrrrcr decode LookupErrorr_has_surrogatesrbr-) r$paramsr5namerMZ value_partsr\rrparamrrrrrFsB      zMimeParameters.paramscCsXg}x8|jD].\}}|r0|jdj|t|q |j|q Wdj|}|rTd|SdS)Nz{}={}z; rsr,)rr'rErr-)r$rrrrrrr.ys zMimeParameters.__str__N)r:r;r<rOrprr.rrrrrBs 3rc@s$eZdZeddZeddZdS)ParameterizedHeaderValuecCs&x t|D]}|jdkr |jSq WiS)Nzmime-parameters)reversedrOr)r$r5rrrrs  zParameterizedHeaderValue.paramscCs4|r*|djdkr*t|dd|dSt|jS)Nr0zmime-parametersrRrRrR)rOr=rM)r$rrrrMszParameterizedHeaderValue.partsN)r:r;r<rprrMrrrrrs rc@seZdZdZdZdZdS) ContentTypez content-typetextplainN)r:r;r<rOmaintypesubtyperrrrrsrc@seZdZdZdZdS)ContentDispositionzcontent-dispositionN)r:r;r<rOcontent_dispositionrrrrrsrc@seZdZdZdZdS)ContentTransferEncodingzcontent-transfer-encoding7bitN)r:r;r<rOrrrrrrsrc@seZdZdZdS) HeaderLabelz header-labelN)r:r;r<rOrrrrrsrc@seZdZdZddZdS)HeaderheadercCsj|jt|jdt|jd|_|djdkr@t|jdnd|_|jd}|r\td|j |dS)NrrQr,zMalformed Header token list) r'rrPr1r#rrOr ValueErrorr4)r$rXrestrrrr4s" z Header._foldN)r:r;r<rOr4rrrrrsrcsveZdZfddZfddZeddZdfdd Zd d Zd d Z ddZ eddZ ddZ ddZ ZS)Terminalcs"tt|j||}||_g|_|S)N)r r__new__rOr>)clsrrOr$)rArrrszTerminal.__new__csdj|jjtt|jS)Nz{}({}))rErAr:r rrF)r$)rArrrFszTerminal.__repr__cCs t|jS)N)r r>)r$rrrrGszTerminal.all_defectsr,cs6dj||jj|jtt|j|js&dn dj|jgS)Nz {}{}/{}({}){}r,z {})rErAr:rOr rrFr>)r$rh)rArrrjs  z Terminal._ppc Cs8t|}y|jd|Stk r2tj||SXdS)Nzus-ascii)rr[rcrZ)r$r\rrrrrr`s  zTerminal.cte_encodecCsdS)Nr)r$rrrrJszTerminal.pop_trailing_wscCsdS)Nr)r$rrrr3szTerminal.pop_leading_fwscCsgS)Nr)r$rrrrUszTerminal.commentscCsdS)NFr)r$rrrrTszTerminal.has_leading_commentcCst||jfS)N)rrO)r$rrr__getnewargs__szTerminal.__getnewargs__)r,)r:r;r<rrFrprGrjr`rJr3rUrTrrqrr)rArrs     rc@s$eZdZeddZddZdZdS)WhiteSpaceTerminalcCsdS)Nrsr)r$rrrrszWhiteSpaceTerminal.valuecCsdS)NTr)r$rrrrIsz!WhiteSpaceTerminal.startswith_fwsTN)r:r;r<rprrIr2rrrrrs rc@s,eZdZeddZddZdZddZdS) ValueTerminalcCs|S)Nr)r$rrrrszValueTerminal.valuecCsdS)NFr)r$rrrrI szValueTerminal.startswith_fwsFcCstjt||S)N)rZr[r)r$r\rrrr_szValueTerminal.as_encoded_wordN)r:r;r<rprrIr2r_rrrrrs rc@s0eZdZeddZeddZddZdZdS) EWWhiteSpaceTerminalcCsdS)Nr,r)r$rrrrszEWWhiteSpaceTerminal.valuecCs |ddS)Nr)r$rrrrszEWWhiteSpaceTerminal.encodedcCsdS)Nr,r)r$rrrr.szEWWhiteSpaceTerminal.__str__TN)r:r;r<rprrr.r2rrrrrs  rr,zlist-separatorrzroute-component-markerz([{}]+)r,z[^{}]+rz\\]z\]z[\x00-\x20\x7F]cCs>t|}|r|jjtj|tj|r:|jjtjddS)z@If input token contains ASCII non-printables, register a defect.z*Non-ASCII characters found in header tokenN)_non_printable_finderr>r'rNonPrintableDefectrrrb)xtextnon_printablesrrr_validate_xtextOs   rc Cstt|d}|dd|ddg\}}g}d}d}xbtt|D]J}||dkrj|rdd}d}nd}qF|rtd}n|||krP|j||qFW|d}dj|dj||dg||fS)akScan printables/quoted-pairs until endchars and return unquoted ptext. This function turns a run of qcontent, ccontent-without-comments, or dtext-with-quoted-printables into a single string by unquoting any quoted printables. It returns the string, the remaining value, and a flag that is True iff there were any quoted printables decoded. r0NFrTr,)r _wsp_splitterrr1r'r-) rendcharsZ _3to2listfragmentrZvcharsescapehad_qpposrrr_get_ptext_to_endcharsYs&   r c Csg}g}d}x|ryt|d\}}}Wn$tk rJ|dd}}}YnX|jdo^|jds|dj||||||fStj|\}}}} |j||j| |}qWdj|||fS)a" Decode a run of RFC2047 encoded words. _decode_ew_run(value) -> (text, value, defects) Scans the supplied value for a run of tokens that look like they are RFC 2047 encoded words, decodes those words into text according to RFC 2047 rules (whitespace between encoded words is discarded), and returns the text and the remaining value (including any leading whitespace on the remaining value), as well as a list of any defects encountered while decoding. The input value may not have any leading whitespace. r,r0z=?z?=) rr startswithrr-rZrr'r&) rr]r>Zlast_wsrr7rr\rZ new_defectsrrr_decode_ew_runxs   rcCs.|j}t|dt|t|d}||fS)zFWS = 1*WSP This isn't the RFC definition. We're using fws to represent tokens where folding can be done, but when we are parsing the *un*folding has already been done so we don't need to watch out for CRLF. NrN)rrr1)rZnewvaluerNrrrget_fwssrc Cst}|jds tjdj|t|ddjdd}|dd|ddg\}}||ddkrvtjdj|dj|}|ddjrt|jdd}|dd|ddg\}}|d|}t |jdkr|j j tj d||_ dj|}ytjd|d\}} } } Wn(tk rBtjd j|j YnX| |_| |_|j j| x|r|d tkrt|\} }|j | q^tt|d} | dd| ddg\}}t|d }t||j |dj|}q^W||fS) zE encoded-word = "=?" charset "?" encoding "?" encoded-text "?=" z=?z"expected encoded word but found {}rNz?=r0r,zwhitespace inside encoded wordz!encoded word format invalid: '{}'rvtext)rrrHeaderParseErrorrEr rr-isdigitr1r>r'rrrZrrr\rr&WSPrrrr)rewZ _3to2list1rrZremstrZ _3to2list3rrr\rr>r5Z _3to2list5charsrrrrget_encoded_wordsN            rc Cs0t}x"|r*|dtkr4t|\}}|j|q |jdryt|\}}Wntjk rdYnrXd}t|dkr|d j dkr|j jtj dd}|rt|dkr|dj d krt |dd|d<|j|q t t|d}|d d|dd g\}}t|d }t||j|d j|}q W|S)aOunstructured = (*([FWS] vchar) *WSP) / obs-unstruct obs-unstruct = *((*LF *CR *(obs-utext) *LF *CR)) / FWS) obs-utext = %d0 / obs-NO-WS-CTL / LF / CR obs-NO-WS-CTL is control characters except WSP/CR/LF. So, basically, we have printable runs, plus control characters or nulls in the obsolete syntax, separated by whitespace. Since RFC 2047 uses the obsolete syntax in its specification, but requires whitespace on either side of the encoded words, I can see no reason to need to separate the non-printable-non-whitespace from the printable runs if they occur, so we parse this into xtext tokens separated by WSP tokens. Because an 'unstructured' value must by definition constitute the entire value, this 'get' routine does not return a remaining value, only the parsed TokenList. rz=?Tr0rNz&missing whitespace before encoded wordFrz encoded-wordNrr,rRrRrR)rwrrr'rrrrr1rOr>rrr rrrr-)rrxr5Zhave_wsZ _3to2list7rrrrrrrys<          rycCs*t|d\}}}t|d}t|||fS)actext = This is not the RFC ctext, since we are handling nested comments in comment and unquoting quoted-pairs here. We allow anything except the '()' characters, but if we find any ASCII other than the RFC defined printable ASCII an NonPrintableDefect is added to the token's defects list. Since quoted pairs are converted to their unquoted values, what is returned is a 'ptext' token. In this case it is a WhiteSpaceTerminal, so it's value is ' '. z()r)r rr)rr_rrr get_qp_ctexts  rcCs*t|d\}}}t|d}t|||fS)aoqcontent = qtext / quoted-pair We allow anything except the DQUOTE character, but if we find any ASCII other than the RFC defined printable ASCII an NonPrintableDefect is added to the token's defects list. Any quoted pairs are converted to their unquoted values, so what is returned is a 'ptext' token. In this case it is a ValueTerminal. rr)r rr)rrrrrr get_qcontents  rcCsNt|}|stjdj||j}|t|d}t|d}t|||fS)zatext = We allow any non-ATOM_ENDS in atext, but add an InvalidATextDefect to the token's defects list if we find non-atext characters. zexpected atext but found '{}'Natext)_non_atom_end_matcherrrrErr1rr)rmrrrr get_atext$s  rcCs|ddkrtjdj|t}|dd}xD|rr|ddkrr|dtkrZt|\}}n t|\}}|j|q0W|s|jjtj d||fS||ddfS)zbare-quoted-string = DQUOTE *([FWS] qcontent) [FWS] DQUOTE A quoted-string without the leading or trailing white space. Its value is the text between the quote marks, with whitespace preserved and quoted pairs decoded. rrzexpected '"' but found '{}'r0Nz"end of header inside quoted string) rrrErrrrr'r>r)rZbare_quoted_stringr5rrrget_bare_quoted_string4s      r cCs|r |ddkr tjdj|t}|dd}x^|r|ddkr|dtkr^t|\}}n&|ddkrxt|\}}n t|\}}|j|q4W|s|j jtj d||fS||ddfS)zcomment = "(" *([FWS] ccontent) [FWS] ")" ccontent = ctext / quoted-pair / comment We handle nested comments here, and quoted-pair in our qp-ctext routine. rrzexpected '(' but found '{}'r0Nrzend of header inside comment) rrrErrr get_commentrr'r>r)rrtr5rrrr!Ls"      r!cCsTt}xD|rJ|dtkrJ|dtkr2t|\}}n t|\}}|j|qW||fS)z,CFWS = (1*([FWS] comment) [FWS]) / FWS r)r CFWS_LEADERrrr!r')rrQr5rrrget_cfwses  r#cCspt}|r,|dtkr,t|\}}|j|t|\}}|j||rh|dtkrht|\}}|j|||fS)zquoted-string = [CFWS] [CFWS] 'bare-quoted-string' is an intermediate class defined by this parser and not by the RFC grammar. It is the quoted string without any attached CFWS. r)rr"r#r'r )rZ quoted_stringr5rrrget_quoted_stringrs      r$cCst}|r,|dtkr,t|\}}|j||rL|dtkrLtjdj|t|\}}|j||r|dtkrt|\}}|j|||fS)z"atom = [CFWS] 1*atext [CFWS] rzexpected atom but found '{}') rr"r#r' ATOM_ENDSrrrEr)rrr5rrrget_atoms       r&cCst}| s|dtkr(tjdj|xP|rx|dtkrxt|\}}|j||r*|ddkr*|jt|dd}q*W|dtkrtjdjd|||fS)z( dot-text = 1*atext *("." 1*atext) rz8expected atom at a start of dot-atom-text but found '{}'rr0Nz4expected atom at end of dot-atom-text but found '{}'rR)rr%rrrErr'r)rZ dot_atom_textr5rrrget_dot_atom_texts     r'cCslt}|dtkr(t|\}}|j|t|\}}|j||rd|dtkrdt|\}}|j|||fS)z- dot-atom = [CFWS] dot-atom-text [CFWS] r)rr"r#r'r')rZdot_atomr5rrr get_dot_atoms       r(cCs|dtkrt|\}}nd}|ddkr8t|\}}n*|dtkrVtjdj|n t|\}}|dk rx|g|dd<||fS)aword = atom / quoted-string Either atom or quoted-string may start with CFWS. We have to peel off this CFWS first to determine which type of word to parse. Afterward we splice the leading CFWS, if any, into the parsed sub-token. If neither an atom or a quoted-string is found before the next special, a HeaderParseError is raised. The token returned is either an Atom or a QuotedString, as appropriate. This means the 'word' level of the formal grammar is not represented in the parse tree; this is because having that extra layer when manipulating the parse tree is more confusing than it is helpful. rNrz1Expected 'atom' or 'quoted-string' but found '{}')r"r#r$SPECIALSrrrEr&)rleaderr5rrrget_words     r+cCst}yt|\}}|j|Wn(tjk rH|jjtjdYnXx|r|dtkr|ddkr|jt|jjtj d|dd}qLyt|\}}WnDtjk r|dt krt |\}}|jjtj dnYnX|j|qLW||fS)a phrase = 1*word / obs-phrase obs-phrase = word *(word / "." / CFWS) This means a phrase can be a sequence of words, periods, and CFWS in any order as long as it starts with at least one word. If anything other than words is detected, an ObsoleteHeaderDefect is added to the token's defect list. We also accept a phrase that starts with CFWS followed by a dot; this is registered as an InvalidHeaderDefect, since it is not supported by even the obsolete grammar. zphrase does not start with wordrrzperiod in 'phrase'r0Nzcomment found without atom) rr+r'rrr>r PHRASE_ENDSrObsoleteHeaderDefectr"r#)rrr5rrr get_phrases.          r.cCstt}d}|dtkr"t|\}}|s6tjdj|yt|\}}Wn^tjk ryt|\}}Wn6tjk r|ddkr|dtkrt }YnXYnX|dk r|g|dd<|j ||o|ddks|dtkr2t t ||\}}|j dkr|jj tjdn|jj tjd||d<y|jjdWn(tk rj|jj tjd YnX||fS) z= local-part = dot-atom / quoted-string / obs-local-part Nrz"expected local-part but found '{}'rzinvalid-obs-local-partzrr-rr[rcNonASCIILocalPartDefect)rrr*r5obs_local_partrrrget_local_partsB         r3c Cst}d}x|o(|ddks,|dtkr*|ddkrl|rN|jjtjd|jtd}|dd}qnD|ddkr|jt|dd |dd}|jjtjd d}q|r|djd kr|jjtjd yt |\}}d}Wn4tj k r|dt kr t |\}}YnX|j|qW|djd ks\|djd krn|djd krn|jjtjd|djd ks|djd kr|djd kr|jjtjd|jrd|_||fS)z' obs-local-part = word *("." word) Frrrzinvalid repeated '.'Tr0Nzmisplaced-specialz/'\' character outside of quoted-string/ccontentrzmissing '.' between wordsrQz!Invalid leading '.' in local partrz"Invalid trailing '.' in local partzinvalid-obs-local-partrRrRrRr) rr,r>r'rrrrrOr+rr"r#)rr2Zlast_non_ws_was_dotr5rrrr0$sV"            r0cCs@t|d\}}}t|d}|r0|jjtjdt|||fS)a dtext = / obs-dtext obs-dtext = obs-NO-WS-CTL / quoted-pair We allow anything except the excluded characters, but if we find any ASCII other than the RFC defined printable ASCII an NonPrintableDefect is added to the token's defects list. Quoted pairs are converted to their unquoted values, so what is returned is a ptext token, in this case a ValueTerminal. If there were quoted-printables, an ObsoleteHeaderDefect is added to the returned token's defect list. z[]rz(quoted printable found in domain-literal)r rr>r'rr-r)rrr rrr get_dtextSs   r4cCs,|rdS|jtjd|jtdddS)NFz"end of input inside domain-literalrzdomain-literal-endT)r'rrr)rdomain_literalrrr_check_for_early_dl_endgs r6cCslt}|dtkr(t|\}}|j||s6tjd|ddkrRtjdj||dd}t||rp||fS|jtdd|dt krt |\}}|j|t |\}}|j|t||r||fS|dt krt |\}}|j|t||r||fS|ddkrtjd j||jtdd |dd}|rd|dtkrdt|\}}|j|||fS) zB domain-literal = [CFWS] "[" *([FWS] dtext) [FWS] "]" [CFWS] rzexpected domain-literal[z6expected '[' at start of domain-literal but found '{}'r0Nzdomain-literal-startrz4expected ']' at end of domain-literal but found '{}'zdomain-literal-end) rr"r#r'rrrEr6rrrr4)rr5r5rrrget_domain_literalosD                    r8c CsZt}d}|dtkr"t|\}}|s6tjdj||ddkrvt|\}}|dk rd|g|dd<|j|||fSyt|\}}Wn"tjk rt |\}}YnX|dk r|g|dd<|j||o|ddkrR|j jtj d|dj dkr|d|dd<xB|rP|ddkrP|jt t |dd\}}|j|qW||fS) z] domain = dot-atom / domain-literal / obs-domain obs-domain = atom *("." atom)) Nrzexpected domain but found '{}'r7rz(domain is not a dot-atom (contains CFWS)zdot-atomr0)rr"r#rrrEr8r'r(r&r>r-rOr)rrr*r5rrr get_domains<         r9cCs~t}t|\}}|j|| s.|ddkrH|jjtjd||fS|jtddt|dd\}}|j|||fS)z( addr-spec = local-part "@" domain rrz"add-spec local part with no domainzaddress-at-symbolr0N)rr3r'r>rrrr9)rrr5rrr get_addr_specs    r:cCst}xf|rl|ddks$|dtkrl|dtkrHt|\}}|j|q|ddkr|jt|dd}qW| s|ddkrtjdj||jtt |dd\}}|j|x|o|ddkrB|jt|dd}|sP|dtkr t|\}}|j||ddkr|jtt |dd\}}|j|qW|sTtjd|ddkrrtjd j||jt dd ||ddfS) z obs-route = obs-domain-list ":" obs-domain-list = *(CFWS / ",") "@" domain *("," [CFWS] ["@" domain]) Returns an obs-route token with the appropriate sub-tokens (that is, there is no obs-domain-list in the parse tree). rrr0Nrz(expected obs-route domain but found '{}'z%end of header while parsing obs-route:z4expected ':' marking end of obs-route but found '{}'zend-of-obs-route-marker) rr"r#r' ListSeparatorrrrERouteComponentMarkerr9r)rZ obs_router5rrr get_obs_routesB               r>cCst}|dtkr(t|\}}|j|| s:|ddkrJtjdj||jtdd|dd}|ddkr|jtdd|jjtj d |dd}||fSyt |\}}Wnztjk r2y"t |\}}|jjtj d Wn(tjk rtjd j|YnX|j|t |\}}YnX|j||r`|ddkr`|dd}n|jjtj d |jtdd|r|dtkrt|\}}|j|||fS) z angle-addr = [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr obs-angle-addr = [CFWS] "<" obs-route addr-spec ">" [CFWS] rzangle-addr-endznull addr-spec in angle-addrz*obsolete route specification in angle-addrz.expected addr-spec or obs-route but found '{}'z"missing trailing '>' on angle-addr) rr"r#r'rrrErr>rr:r>r-)rZ angle_addrr5rrrget_angle_addrsJ                rAcCs<t}t|\}}|j|dd|jdd|_||fS)z display-name = phrase Because this is simply a name-rule, we don't return a display-name token containing a phrase, but rather a display-name token with the content of the phrase. N)rr.r&r>)rrr5rrrget_display_name!s  rBcCst}d}|dtkr6t|\}}|s6tjdj||ddkr|dtkr^tjdj|t|\}}|s~tjdj||dk r|g|ddd<d}|j|t |\}}|dk r|g|dd<|j|||fS)z, name-addr = [display-name] angle-addr Nrz!expected name-addr but found '{}'r?) rr"r#rrrEr,rBr'rA)rZ name_addrr*r5rrr get_name_addr0s0           rCcCst}yt|\}}WnNtjk rdyt|\}}Wn&tjk r^tjdj|YnXYnXtdd|jDrd|_|j |||fS)z& mailbox = name-addr / addr-spec zexpected mailbox but found '{}'css|]}t|tjVqdS)N)rarr)rBrCrrrrD]szget_mailbox..zinvalid-mailbox) rrCrrr:rErdrGrOr')rrr5rrr get_mailboxNs  rDcCsht}xX|r^|d|kr^|dtkrF|jt|dd|dd}qt|\}}|j|qW||fS)z Read everything up to one of the chars in endchars. This is outside the formal grammar. The InvalidMailbox TokenList that is returned acts like a Mailbox, but the data attributes are None. rzmisplaced-specialr0N)rr,r'rr.)rrZinvalid_mailboxr5rrrget_invalid_mailboxcs   rEc Cst}x|o|ddkryt|\}}|j|Wntjk r@d}|dtkrt|\}}| sz|ddkr|j||jjtjdn@t |d\}}|dk r|g|dd<|j||jjtj dnb|ddkr|jjtjdnBt |d\}}|dk r |g|dd<|j||jjtj dYnX|r|ddkr|d }d |_ t |d\}}|j ||jjtj d|r |ddkr |jt |dd}q W||fS) aJ mailbox-list = (mailbox *("," mailbox)) / obs-mbox-list obs-mbox-list = *([CFWS] ",") mailbox *("," [mailbox / CFWS]) For this routine we go outside the formal grammar in order to improve error handling. We recognize the end of the mailbox list only at the end of the value or at a ';' (the group terminator). This is so that we can turn invalid mailboxes into InvalidMailbox tokens and continue parsing any remaining valid mailboxes. We also allow all mailbox entries to be null, and this condition is handled appropriately at a higher level. r;Nz,;zempty element in mailbox-listzinvalid mailbox in mailbox-listrr0zinvalid-mailboxrR)rrDr'rrr"r#r>r-rErrOr&r<)rZ mailbox_listr5r*rrrrget_mailbox_listusN                   rGcCst}|s$|jjtjd||fSd}|r|dtkrt|\}}|sl|jjtjd|j|||fS|ddkr|j|||fSt|\}}t|j dkr|dk r|j||j ||jjtj d||fS|dk r|g|dd<|j|||fS)zg group-list = mailbox-list / CFWS / obs-group-list obs-group-list = 1*([CFWS] ",") [CFWS] zend of header before group-listNrzend of header in group-listrFzgroup-list with empty entries) rr>r'rrr"r#rGr1rr&r-)rZ group_listr*r5rrrget_group_lists8           rHcCst}t|\}}| s$|ddkr4tjdj||j||jtdd|dd}|r|ddkr|jtdd||ddfSt|\}}|j||s|jjtj d |ddkrtjd j||jtdd|dd}|o|dt krt |\}}|j|||fS) z7 group = display-name ":" [group-list] ";" [CFWS] rr;z8expected ':' at end of group display name but found '{}'zgroup-display-name-terminatorr0NrFzgroup-terminatorzend of header in groupz)expected ';' at end of group but found {}) rrBrrrEr'rrHr>rr"r#)rrr5rrr get_groups2           rIcCsxt}yt|\}}WnNtjk rdyt|\}}Wn&tjk r^tjdj|YnXYnX|j|||fS)a address = mailbox / group Note that counter-intuitively, an address can be either a single address or a list of addresses (a group). This is why the returned Address object has a 'mailboxes' attribute which treats a single address as a list of length one. When you need to differentiate between to two cases, extract the single element, which is either a mailbox or a group token. zexpected address but found '{}')rrIrrrDrEr')rrr5rrr get_addresss rJcCst}x|ryt|\}}|j|Wn$tjk rP}zd}|dtkrt|\}}| sr|ddkr|j||jjtjdnFt |d\}}|dk r|g|dd<|jt |g|jjtj dnh|ddkr|jjtjdnHt |d\}}|dk r|g|dd<|jt |g|jjtj dWYdd}~XnX|r|ddkr|d d}d|_ t |d\}}|j ||jjtj d|r |jtdd |dd}q W||fS) a address_list = (address *("," address)) / obs-addr-list obs-addr-list = *([CFWS] ",") address *("," [address / CFWS]) We depart from the formal grammar here by continuing to parse until the end of the input, assuming the input to be entirely composed of an address-list. This is always true in email parsing, and allows us to skip invalid addresses to parse additional valid ones. Nrrz"address-list entry with no contentzinvalid address in address-listzempty element in address-listr0zinvalid-mailboxzlist-separatorrR)rrJr'rrr"r#r>r-rErrrOr&r)rZ address_listr5errr*rrrrget_address_list sN                  rLcCst}|s |jjtjd|S|dtkrXt|\}}|j||sX|jjtjdd}x8|r|ddkr|dtkr||d7}|dd}q^W|js|jjtjdj ||jt |d nt ||_ |jt |d |o|dtkr t|\}}|j|| s |ddkrX|j dk r>|jjtjd |rT|jt |d |S|jt dd |dd}|r|dtkrt|\}}|j||s|j dk r|jjtjd |Sd}x2|r|dtkr||d7}|dd}qW|js2|jjtjd j ||jt |d nt ||_ |jt |d |rv|dtkrvt|\}}|j||r|jjtjd|jt |d |S)zE mime-version = [CFWS] 1*digit [CFWS] "." [CFWS] 1*digit [CFWS] z%Missing MIME version number (eg: 1.0)rz0Expected MIME version number but found only CFWSr,rr0Nz1Expected MIME major version number but found {!r}rdigitsz0Incomplete MIME version; found only major numberzversion-separatorz1Expected MIME minor version number but found {!r}z'Excess non-CFWS text after MIME version)rr>r'rHeaderMissingRequiredValuer"r#rrrErrrr)rZ mime_versionr5rMrrrparse_mime_versionL sv                        rOcCsht}xX|r^|ddkr^|dtkrF|jt|dd|dd}qt|\}}|j|qW||fS)z Read everything up to the next ';'. This is outside the formal grammar. The InvalidParameter TokenList that is returned acts like a Parameter, but the data attributes are None. rrFzmisplaced-specialr0N)rr,r'rr.)rZinvalid_parameterr5rrrget_invalid_parameter s   rPcCsNt|}|stjdj||j}|t|d}t|d}t|||fS)a8ttext = We allow any non-TOKEN_ENDS in ttext, but add defects to the token's defects list if we find non-ttext characters. We also register defects for *any* non-printables even though the RFC doesn't exclude all of them, because we follow the spirit of RFC 5322. zexpected ttext but found '{}'Nttext)_non_token_end_matcherrrrErr1rr)rrrQrrr get_ttext s   rScCst}|r,|dtkr,t|\}}|j||rL|dtkrLtjdj|t|\}}|j||r|dtkrt|\}}|j|||fS)ztoken = [CFWS] 1*ttext [CFWS] The RFC equivalent of ttext is any US-ASCII chars except space, ctls, or tspecials. We also exclude tabs even though the RFC doesn't. The RFC implies the CFWS but is not explicit about it in the BNF. rzexpected token but found '{}') rr"r#r' TOKEN_ENDSrrrErS)rZmtokenr5rrr get_token s        rUcCsNt|}|stjdj||j}|t|d}t|d}t|||fS)aQattrtext = 1*(any non-ATTRIBUTE_ENDS character) We allow any non-ATTRIBUTE_ENDS in attrtext, but add defects to the token's defects list if we find non-attrtext characters. We also register defects for *any* non-printables even though the RFC doesn't exclude all of them, because we follow the spirit of RFC 5322. z expected attrtext but found {!r}Nr)_non_attribute_end_matcherrrrErr1rr)rrrrrr get_attrtext s   rWcCst}|r,|dtkr,t|\}}|j||rL|dtkrLtjdj|t|\}}|j||r|dtkrt|\}}|j|||fS)aH [CFWS] 1*attrtext [CFWS] This version of the BNF makes the CFWS explicit, and as usual we use a value terminal for the actual run of characters. The RFC equivalent of attrtext is the token characters, with the subtraction of '*', "'", and '%'. We include tab in the excluded set just as we do for token. rzexpected token but found '{}') rr"r#r'ATTRIBUTE_ENDSrrrErW)rrr5rrr get_attribute s        rYcCsNt|}|stjdj||j}|t|d}t|d}t|||fS)zattrtext = 1*(any non-ATTRIBUTE_ENDS character plus '%') This is a special parsing routine so that we get a value that includes % escapes as a single string (which we decode as a single string later). z)expected extended attrtext but found {!r}Nzextended-attrtext)#_non_extended_attribute_end_matcherrrrErr1rr)rrrrrrget_extended_attrtext s  r[cCst}|r,|dtkr,t|\}}|j||rL|dtkrLtjdj|t|\}}|j||r|dtkrt|\}}|j|||fS)z [CFWS] 1*extended_attrtext [CFWS] This is like the non-extended version except we allow % characters, so that we can pick up an encoded value as a single string. rzexpected token but found '{}') rr"r#r'EXTENDED_ATTRIBUTE_ENDSrrrEr[)rrr5rrrget_extended_attribute s       r]cCst}| s|ddkr(tjdj||jtdd|dd}| sX|dj rhtjdj|d}x,|r|djr||d7}|dd}qnW|dd kr|d kr|jjtjd t ||_ |jt|d ||fS) a6 '*' digits The formal BNF is more complicated because leading 0s are not allowed. We check for that and add a defect. We also assume no CFWS is allowed between the '*' and the digits, though the RFC is not crystal clear on that. The caller should already have dealt with leading CFWS. r*zExpected section but found {}zsection-markerr0Nz$Expected section number but found {}r,0z§ion numberhas an invalid leading 0rM) rrrrEr'rrr>InvalidHeaderErrorrr)rrrMrrr get_section s$    racCst}|stjdd}|dtkr0t|\}}|sDtjdj||ddkr^t|\}}n t|\}}|dk r|g|dd<|j|||fS)z quoted-string / attribute z&Expected value but found end of stringNrz Expected value but found only {}r) rrrr"r#rEr$r]r')rvr*r5rrr get_value< s       rcc Cst}t|\}}|j|| s.|ddkrN|jjtjdj|||fS|ddkry t|\}}d|_|j|Wntj k rYnX|stj d|ddkr|jt dd|dd }d|_ |dd krtj d |jt d d |dd }d }|r.|dt kr.t |\}}|j|d }|}|j rH|rH|dd krHt|\}}|j}d}|jdkr|r|ddkrd}n$t|\}} | r| ddkrd}n(yt|\}} Wn Yn X| sd}|r2|jjtjd|j|x,|D]$} | jdkrg| d d <| }PqW|}nd }|jjtjd|rb|ddkrbd }n t|\}}|j s|jdkr| s|ddkr|j||d k r| st||}||fS|jjtjd|s|jjtjd|j||d kr||fSn|d k rVx|D]} | jdkr"Pq"W| jdk|j| | j|_|ddkrttj dj||jt dd|dd }|r|ddkrt|\}}|j||j|_| s|ddkrtj dj||jt dd|dd }|d k rZt} x>|rR|dtkr8t|\}}n t|\}}| j|qW| }n t|\}}|j||d k r| st||}||fS)aY attribute [section] ["*"] [CFWS] "=" value The CFWS is implied by the RFC but not made explicit in the BNF. This simplified form of the BNF from the RFC is made to conform with the RFC BNF through some extra checks. We do it this way because it makes both error recovery and working with the resulting parse tree easier. rrFz)Parameter contains name ({}) but no valuer^TzIncomplete parameterzextended-parameter-markerr0N=zParameter not followed by '='zparameter-separatorrF'z5Quoted string value for extended parameter is invalidzbare-quoted-stringzZParameter marked as extended but appears to have a quoted string value that is non-encodedzcApparent initial-extended-value but attribute was not marked as extended or was not initial sectionz(Missing required charset/lang delimiterszextended-attrtextrz=Expected RFC2231 char/lang encoding delimiter, but found {!r}zRFC2231 delimiterz;Expected RFC2231 char/lang encoding delimiter, but found {})rrYr'r>rrrErarrrrr"r#r$rrrWr[rOrcAssertionErrorrr\rrrrr) rrr5r*rZappendtoZqstringZ inner_valueZ semi_validrtrbrrr get_parameterR s                                             rhcCsht}xZ|rbyt|\}}|j|Wntjk r}zd}|dtkrZt|\}}|sl|j||S|ddkr|dk r|j||jjtjdn@t |\}}|r|g|dd<|j||jjtjdj |WYdd}~XnX|r@|ddkr@|d }d|_ t |\}}|j ||jjtjdj ||r |jt dd |dd}q W|S) a! parameter *( ";" parameter ) That BNF is meant to indicate this routine should only be called after finding and handling the leading ';'. There is no corresponding rule in the formal RFC grammar, but it is more convenient for us for the set of parameters to be treated as its own TokenList. This is 'parse' routine because it consumes the reminaing value, but it would never be called to parse a full header. Instead it is called to parse everything after the non-parameter value of a specific MIME header. NrrFzparameter entry with no contentzinvalid parameter {!r}r0zinvalid-parameterz)parameter with invalid trailing text {!r}zparameter-separatorrR)rrhr'rrr"r#r>rrPrErOr&r)rZmime_parametersr5rKr*rrrrparse_mime_parameters sD                 ricCsxX|rX|ddkrX|dtkr@|jt|dd|dd}qt|\}}|j|qW|sbdS|jtdd|jt|dddS)zBDo our best to find the parameters in an invalid MIME header rrFzmisplaced-specialr0Nzparameter-separator)r,r'rr.ri) tokenlistrr5rrr_find_mime_parameters s  rkcCst}d}|s$|jjtjd|Syt|\}}Wn8tjk rl|jjtjdj|t |||SX|j|| s|ddkr|jjtjd|rt |||S|j j j |_ |jtdd|dd }yt|\}}Wn:tjk r$|jjtjd j|t |||SX|j||j j j |_|sJ|S|dd kr|jjtjd j||` |`t |||S|jtd d |jt|dd |S)z maintype "/" subtype *( ";" parameter ) The maintype and substype are tokens. Theoretically they could be checked against the official IANA list + x-token, but we don't do that. Fz"Missing content type specificationz(Expected content maintype but found {!r}r/zInvalid content typezcontent-type-separatorr0Nz'Expected content subtype but found {!r}rFzr'rrNrUrrrErkrrlowerrrrri)rctypeZrecoverr5rrrparse_content_type_header! sX             roc Cst}|s |jjtjd|Syt|\}}Wn8tjk rhtjjtjdj |t |||SX|j||j j j |_|s|S|ddkr|jjtjdj |t |||S|jtdd|jt|dd|S) z* disposition-type *( ";" parameter ) zMissing content dispositionz+Expected content disposition but found {!r}rrFzCOnly parameters are valid after content disposition, but found {!r}zparameter-separatorr0N)rr>r'rrNrUrrnrrErkrrrmrrri)rZ disp_headerr5rrr parse_content_disposition_headerZ s2        rpc Cst}|s |jjtjd|Syt|\}}Wn.tjk r^tjjtjdj |YnX|j||j j j |_ |s|Sx^|r|jjtjd|dtkr|jt|dd|dd}qt|\}}|j|qW|S)z mechanism z!Missing content transfer encodingz1Expected content trnasfer encoding but found {!r}z*Extra text after content transfer encodingrzmisplaced-specialr0N)rr>r'rrNrUrrnrrErrrmrr,rr.)rZ cte_headerr5rrr&parse_content_transfer_encoding_headerx s.      rq)__doc__ __future__rrrrZfuture.builtinsrrrr r re collectionsr r Zfuture.backports.urllib.parser rZfuture.backports.emailrrZrrrrr"r)r%rr,Z TSPECIALSrTZ ASPECIALSrXr\robjectrr=rrrwrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr<r=compilerEr-rrrmatchrfindallrrRrVrZrr rrrryrrrr r!r#r$r&r'r(r+r.r3r0r4r6r8r9r:r>rArBrCrDrErGrHrIrJrLrOrPrSrUrWrYr[r]rarcrhrirkrorprqrrrrDs            TD N_     '# B3           ,7 &'/'$).9%>D 49