관리-도구
편집 파일: _tokenizer.cpython-36.pyc
3 �Re�, � @ s� d dl mZmZmZ d dlmZ d dlmZm Z d dl m Z ddl m Z ddl mZ ddl mZmZ dd l mZmZmZ dd l mZmZ dd l mZ dd lmZ dd lmZ ee�Ze dkr�e Z ne Z G dd� de �Z dS )� )�absolute_import�division�unicode_literals)�unichr)�deque� OrderedDict)� version_info� )�spaceCharacters)�entities)� asciiLetters�asciiUpper2Lower)�digits� hexDigits�EOF)� tokenTypes� tagTokenTypes)�replacementCharacters)�HTMLInputStream)�Trie� � c sd e Zd ZdZd�� fdd� Zdd� Zdd� Zd�d d �Zd d � Zdd� Z dd� Z dd� Z dd� Z dd� Z dd� Zdd� Zd d � Zd d � Zd d!� Zd"d#� Zd$d%� Zd&d'� Zd(d)� Zd*d+� Zd,d-� Zd.d/� Zd0d1� Zd2d3� Zd4d5� Z d6d7� Z d8d9� Z d:d;� Z d<d=� Z d>d?� Z!d@dA� Z"dBdC� Z#dDdE� Z$dFdG� Z%dHdI� Z&dJdK� Z'dLdM� Z(dNdO� Z)dPdQ� Z*dRdS� Z+dTdU� Z,dVdW� Z-dXdY� Z.dZd[� Z/d\d]� Z0d^d_� Z1d`da� Z2dbdc� Z3ddde� Z4dfdg� Z5dhdi� Z6djdk� Z7dldm� Z8dndo� Z9dpdq� Z:drds� Z;dtdu� Z<dvdw� Z=dxdy� Z>dzd{� Z?d|d}� Z@d~d� ZAd�d�� ZBd�d�� ZCd�d�� ZDd�d�� ZEd�d�� ZFd�d�� ZGd�d�� ZHd�d�� ZId�d�� ZJd�d�� ZKd�d�� ZL� ZMS )�� HTMLTokenizera This class takes care of tokenizing HTML. * self.currentToken Holds the token that is currently being processed. * self.state Holds a reference to the method to be invoked... XXX * self.stream Points to HTMLInputStream object. Nc sF t |f|�| _|| _d| _g | _| j| _d| _d | _t t | �j � d S )NF) r �stream�parser� escapeFlag� lastFourChars� dataState�state�escape� currentToken�superr �__init__)�selfr r �kwargs)� __class__� ��/builddir/build/BUILDROOT/alt-python36-pip-20.2.4-5.el8.x86_64/opt/alt/python36/lib/python3.6/site-packages/pip/_vendor/html5lib/_tokenizer.pyr" ( s zHTMLTokenizer.__init__c c s\ t g �| _xL| j� rVx&| jjr:td | jjjd�d�V qW x| jrR| jj� V q>W q W dS )z� This is where the magic happens. We do our usually processing through the states and when we have a token to return we yield the token which pauses processing until the next token is requested. � ParseErrorr )�type�dataN)r � tokenQueuer r �errorsr �pop�popleft)r# r& r& r' �__iter__7 s zHTMLTokenizer.__iter__c % C s( t }d}|rt}d}g }| jj� }x(||krJ|tk rJ|j|� | jj� }q$W tdj|�|�}|tkr�t| }| j jt d dd|id�� �nld| ko�d kn s�|d kr�d }| j jt d dd|id�� �n(d | ko�d kn �s�d| ko�dkn �s�d| k�odkn �s�d| k�o4dkn �s�|t ddddddddd d d d d d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d g#�k�r�| j jt d dd|id�� y t |�}W n> t k �r� |d6 }t d|d? B �t d7|d8@ B � }Y nX |d9k�r$| j jt d d:d;�� | jj|� |S )<z�This function returns either U+FFFD or the character based on the decimal or hexadecimal representation. It also discards ";" if present. If not present self.tokenQueue.append({"type": tokenTypes["ParseError"]}) is invoked. � � � r( z$illegal-codepoint-for-numeric-entity� charAsInt)r) r* �datavarsi � i�� i�� u �r � � � � � i� i� � i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i i � i� �;z numeric-entity-without-semicolon)r) r* )r r r �charr �append�int�joinr r+ r � frozenset�chr� ValueError�unget) r# �isHex�allowed�radix� charStack�cr3 r< �vr& r& r' �consumeNumberEntityG s` & z!HTMLTokenizer.consumeNumberEntityFc C s� d}| j j� g}|d tksB|d tddfksB|d k rV||d krV| j j|d � �n"|d dk�rd}|j| j j� � |d dkr�d }|j| j j� � |r�|d tks�| r�|d tkr�| j j|d � | j|�}n4| j jt d d d �� | j j|j � � dd j |� }�njx8|d tk �rFt jd j |���s2P |j| j j� � �qW y$t jd j |d d� ��}t|�}W n tk �r� d }Y nX |d k �rD|d dk�r�| j jt d dd �� |d dk�r|�r|| tk�s�|| tk�s�|| dk�r| j j|j � � dd j |� }n.t| }| j j|j � � |d j ||d � �7 }n4| j jt d dd �� | j j|j � � dd j |� }|�r�| jd d d |7 < n*|tk�r�d}nd}| j jt | |d �� d S ) N�&r �<�#Fr �x�XTr( zexpected-numeric-entity)r) r* r2 r; z named-entity-without-semicolon�=zexpected-named-entityr* �SpaceCharacters� Characters���)rN rO rS rS rS rS rS rS rS rS )r r< r r rC r= r r rJ r+ r r- r? � entitiesTrie�has_keys_with_prefix�longest_prefix�len�KeyErrorr r r ) r# � allowedChar� fromAttribute�outputrG �hex� entityName� entityLength� tokenTyper& r& r' � consumeEntity� sf zHTMLTokenizer.consumeEntityc C s | j |dd� dS )zIThis method replaces the need for "entityInAttributeValueState". T)rY rZ N)r` )r# rY r& r& r' �processEntityInAttribute� s z&HTMLTokenizer.processEntityInAttributec C s� | j }|d tkr�|d jt�|d< |d td krp|d }t|�}t|�t|�krh|j|ddd � � ||d< |d td kr�|d r�| jj td d d �� |d r�| jj td d d �� | jj |� | j | _ dS )z�This method is a generic handler for emitting the tags. It also sets the state to "data" because that's what's needed after a token has been emitted. r) �name�StartTagr* Nr �EndTagr( zattributes-in-end-tag)r) r* � selfClosingz self-closing-flag-on-end-tagrS ) r r � translater r � attributeMaprW �updater+ r= r r )r# �token�rawr* r& r& r' �emitCurrentToken� s$ z HTMLTokenizer.emitCurrentTokenc C s� | j j� }|dkr | j| _n�|dkr.| j| _n�|dkrd| jjtd dd�� | jjtd dd�� n`|tkrpdS |t kr�| jjtd || j j t d � d�� n&| j j d �}| jjtd || d�� d S ) NrK rL � r( zinvalid-codepoint)r) r* rR FrQ T)rK rL rl ) r r< �entityDataStater � tagOpenStater+ r= r r r � charsUntil)r# r* �charsr& r& r' r s& zHTMLTokenizer.dataStatec C s | j � | j| _dS )NT)r` r r )r# r& r& r' rm s z HTMLTokenizer.entityDataStatec C s� | j j� }|dkr | j| _n�|dkr.| j| _n�|tkr:dS |dkrp| jjtd dd�� | jjtd d d�� nT|t kr�| jjtd || j j t d � d�� n&| j j d �}| jjtd || d�� d S ) NrK rL Frl r( zinvalid-codepoint)r) r* rR u �rQ T)rK rL rl ) r r< �characterReferenceInRcdatar �rcdataLessThanSignStater r+ r= r r ro )r# r* rp r& r& r' � rcdataState" s& zHTMLTokenizer.rcdataStatec C s | j � | j| _dS )NT)r` rs r )r# r& r& r' rq ? s z(HTMLTokenizer.characterReferenceInRcdatac C s� | j j� }|dkr | j| _nh|dkrR| jjtd dd�� | jjtd dd�� n2|tkr^dS | j jd �}| jjtd || d�� d S ) NrL rl r( zinvalid-codepoint)r) r* rR u �FT)rL rl ) r r< �rawtextLessThanSignStater r+ r= r r ro )r# r* rp r& r& r' � rawtextStateD s zHTMLTokenizer.rawtextStatec C s� | j j� }|dkr | j| _nh|dkrR| jjtd dd�� | jjtd dd�� n2|tkr^dS | j jd �}| jjtd || d�� d S ) NrL rl r( zinvalid-codepoint)r) r* rR u �FT)rL rl ) r r< �scriptDataLessThanSignStater r+ r= r r ro )r# r* rp r& r& r' �scriptDataStateV s z HTMLTokenizer.scriptDataStatec C sr | j j� }|tkrdS |dkrL| jjtd dd�� | jjtd dd�� n"| jjtd || j jd� d�� dS ) NFrl r( zinvalid-codepoint)r) r* rR u �T)r r< r r+ r= r ro )r# r* r& r& r' �plaintextStateh s z HTMLTokenizer.plaintextStatec C s | j j� }|dkr | j| _n�|dkr.| j| _n�|tkrVtd |g ddd�| _| j| _n�|dkr�| j j td dd �� | j j td d d �� | j | _nt|d kr�| j j td d d �� | j j |� | j | _n@| j j td dd �� | j j td dd �� | j j |� | j | _dS )N�!�/rc F)r) rb r* re �selfClosingAcknowledged�>r( z'expected-tag-name-but-got-right-bracket)r) r* rR z<>�?z'expected-tag-name-but-got-question-markzexpected-tag-namerL T)r r< �markupDeclarationOpenStater �closeTagOpenStater r r � tagNameStater+ r= r rC �bogusCommentState)r# r* r& r& r' rn w s6 zHTMLTokenizer.tagOpenStatec C s� | j j� }|tkr0td |g dd�| _| j| _n�|dkrX| jjtd dd�� | j | _nn|t kr�| jjtd dd�� | jjtd d d�� | j | _n0| jjtd d d |id �� | j j |� | j | _dS )Nrd F)r) rb r* re r| r( z*expected-closing-tag-but-got-right-bracket)r) r* z expected-closing-tag-but-got-eofrR z</z!expected-closing-tag-but-got-charr* )r) r* r4 T) r r< r r r r� r r+ r= r r rC r� )r# r* r& r& r' r � s( z HTMLTokenizer.closeTagOpenStatec C s� | j j� }|tkr | j| _n�|dkr.| j� n~|tkrV| jjt d dd�� | j | _nV|dkrh| j | _nD|dkr�| jjt d dd�� | j d d 7 < n| j d |7 < d S ) Nr| r( zeof-in-tag-name)r) r* rz rl zinvalid-codepointrb u �T) r r< r �beforeAttributeNameStater rk r r+ r= r r �selfClosingStartTagStater )r# r* r& r& r' r� � s" zHTMLTokenizer.tagNameStatec C sP | j j� }|dkr"d| _| j| _n*| jjtd dd�� | j j|� | j | _dS )Nrz r2 rR rL )r) r* T) r r< �temporaryBuffer�rcdataEndTagOpenStater r+ r= r rC rs )r# r* r& r& r' rr � s z%HTMLTokenizer.rcdataLessThanSignStatec C sX | j j� }|tkr*| j|7 _| j| _n*| jjtd dd�� | j j |� | j | _dS )NrR z</)r) r* T) r r< r r� �rcdataEndTagNameStater r+ r= r rC rs )r# r* r& r& r' r� � s z#HTMLTokenizer.rcdataEndTagOpenStatec C s | j o | j d j� | jj� k}| jj� }|tkrT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | j � | j | _nH|t kr�| j|7 _n0| j jtd d| j d �� | jj|� | j| _d S ) Nrb rd F)r) rb r* re rz r| rR z</)r) r* T)r �lowerr� r r< r r r� r r� rk r r r+ r= rC rs )r# � appropriater* r& r& r' r� � s2 z#HTMLTokenizer.rcdataEndTagNameStatec C sP | j j� }|dkr"d| _| j| _n*| jjtd dd�� | j j|� | j | _dS )Nrz r2 rR rL )r) r* T) r r< r� �rawtextEndTagOpenStater r+ r= r rC ru )r# r* r& r& r' rt � s z&HTMLTokenizer.rawtextLessThanSignStatec C sX | j j� }|tkr*| j|7 _| j| _n*| jjtd dd�� | j j |� | j | _dS )NrR z</)r) r* T) r r< r r� �rawtextEndTagNameStater r+ r= r rC ru )r# r* r& r& r' r� s z$HTMLTokenizer.rawtextEndTagOpenStatec C s | j o | j d j� | jj� k}| jj� }|tkrT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | j � | j | _nH|t kr�| j|7 _n0| j jtd d| j d �� | jj|� | j| _d S ) Nrb rd F)r) rb r* re rz r| rR z</)r) r* T)r r� r� r r< r r r� r r� rk r r r+ r= rC ru )r# r� r* r& r& r' r� s2 z$HTMLTokenizer.rawtextEndTagNameStatec C sx | j j� }|dkr"d| _| j| _nR|dkrJ| jjtd dd�� | j| _n*| jjtd dd�� | j j |� | j | _dS ) Nrz r2 ry rR z<!)r) r* rL T) r r< r� �scriptDataEndTagOpenStater r+ r= r �scriptDataEscapeStartStaterC rw )r# r* r& r& r' rv , s z)HTMLTokenizer.scriptDataLessThanSignStatec C sX | j j� }|tkr*| j|7 _| j| _n*| jjtd dd�� | j j |� | j | _dS )NrR z</)r) r* T) r r< r r� �scriptDataEndTagNameStater r+ r= r rC rw )r# r* r& r& r' r� : s z'HTMLTokenizer.scriptDataEndTagOpenStatec C s | j o | j d j� | jj� k}| jj� }|tkrT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | j � | j | _nH|t kr�| j|7 _n0| j jtd d| j d �� | jj|� | j| _d S ) Nrb rd F)r) rb r* re rz r| rR z</)r) r* T)r r� r� r r< r r r� r r� rk r r r+ r= rC rw )r# r� r* r& r& r' r� E s2 z'HTMLTokenizer.scriptDataEndTagNameStatec C sJ | j j� }|dkr2| jjtd dd�� | j| _n| j j|� | j| _dS )N�-rR )r) r* T) r r< r+ r= r � scriptDataEscapeStartDashStater rC rw )r# r* r& r& r' r� a s z(HTMLTokenizer.scriptDataEscapeStartStatec C sJ | j j� }|dkr2| jjtd dd�� | j| _n| j j|� | j| _dS )Nr� rR )r) r* T) r r< r+ r= r � scriptDataEscapedDashDashStater rC rw )r# r* r& r& r' r� k s z,HTMLTokenizer.scriptDataEscapeStartDashStatec C s� | j j� }|dkr2| jjtd dd�� | j| _n�|dkrD| j| _nn|dkrz| jjtd dd�� | jjtd dd�� n8|tkr�| j | _n&| j j d �}| jjtd || d�� d S ) Nr� rR )r) r* rL rl r( zinvalid-codepointu �T)rL r� rl ) r r< r+ r= r �scriptDataEscapedDashStater �"scriptDataEscapedLessThanSignStater r ro )r# r* rp r& r& r' �scriptDataEscapedStateu s" z$HTMLTokenizer.scriptDataEscapedStatec C s� | j j� }|dkr2| jjtd dd�� | j| _n�|dkrD| j| _nn|dkr�| jjtd dd�� | jjtd dd�� | j| _n0|t kr�| j | _n | jjtd |d�� | j| _d S ) Nr� rR )r) r* rL rl r( zinvalid-codepointu �T) r r< r+ r= r r� r r� r� r r )r# r* r& r& r' r� � s" z(HTMLTokenizer.scriptDataEscapedDashStatec C s� | j j� }|dkr*| jjtd dd�� n�|dkr<| j| _n�|dkrd| jjtd dd�� | j| _nn|dkr�| jjtd dd�� | jjtd d d�� | j| _n0|t kr�| j | _n | jjtd |d�� | j| _d S ) Nr� rR )r) r* rL r| rl r( zinvalid-codepointu �T) r r< r+ r= r r� r rw r� r r )r# r* r& r& r' r� � s& z,HTMLTokenizer.scriptDataEscapedDashDashStatec C s� | j j� }|dkr"d| _| j| _n\|tkrT| jjtd d| d�� || _| j | _n*| jjtd dd�� | j j |� | j | _dS )Nrz r2 rR rL )r) r* T) r r< r� � scriptDataEscapedEndTagOpenStater r r+ r= r � scriptDataDoubleEscapeStartStaterC r� )r# r* r& r& r' r� � s z0HTMLTokenizer.scriptDataEscapedLessThanSignStatec C sP | j j� }|tkr"|| _| j| _n*| jjtd dd�� | j j |� | j | _dS )NrR z</)r) r* T) r r< r r� � scriptDataEscapedEndTagNameStater r+ r= r rC r� )r# r* r& r& r' r� � s z.HTMLTokenizer.scriptDataEscapedEndTagOpenStatec C s | j o | j d j� | jj� k}| jj� }|tkrT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | j � | j | _nH|t kr�| j|7 _n0| j jtd d| j d �� | jj|� | j| _d S ) Nrb rd F)r) rb r* re rz r| rR z</)r) r* T)r r� r� r r< r r r� r r� rk r r r+ r= rC r� )r# r� r* r& r& r' r� � s2 z.HTMLTokenizer.scriptDataEscapedEndTagNameStatec C s� | j j� }|ttd�B krR| jjtd |d�� | jj� dkrH| j | _ q�| j | _ nB|t kr�| jjtd |d�� | j|7 _n| j j |� | j | _ dS )Nrz r| rR )r) r* �scriptT)rz r| )r r< r r@ r+ r= r r� r� � scriptDataDoubleEscapedStater r� r rC )r# r* r& r& r' r� � s z.HTMLTokenizer.scriptDataDoubleEscapeStartStatec C s� | j j� }|dkr2| jjtd dd�� | j| _n�|dkrZ| jjtd dd�� | j| _nt|dkr�| jjtd dd�� | jjtd dd�� n>|tkr�| jjtd d d�� | j | _n| jjtd |d�� d S ) Nr� rR )r) r* rL rl r( zinvalid-codepointu �zeof-in-script-in-scriptT) r r< r+ r= r � scriptDataDoubleEscapedDashStater �(scriptDataDoubleEscapedLessThanSignStater r )r# r* r& r& r' r� � s$ z*HTMLTokenizer.scriptDataDoubleEscapedStatec C s� | j j� }|dkr2| jjtd dd�� | j| _n�|dkrZ| jjtd dd�� | j| _n�|dkr�| jjtd dd�� | jjtd dd�� | j| _nF|t kr�| jjtd d d�� | j | _n | jjtd |d�� | j| _d S ) Nr� rR )r) r* rL rl r( zinvalid-codepointu �zeof-in-script-in-scriptT) r r< r+ r= r �$scriptDataDoubleEscapedDashDashStater r� r� r r )r# r* r& r& r' r� s( z.HTMLTokenizer.scriptDataDoubleEscapedDashStatec C s | j j� }|dkr*| jjtd dd�� n�|dkrR| jjtd dd�� | j| _n�|dkrz| jjtd dd�� | j| _n�|dkr�| jjtd dd�� | jjtd d d�� | j| _nF|t kr�| jjtd d d�� | j | _n | jjtd |d�� | j| _d S ) Nr� rR )r) r* rL r| rl r( zinvalid-codepointu �zeof-in-script-in-scriptT) r r< r+ r= r r� r rw r� r r )r# r* r& r& r' r� % s, z2HTMLTokenizer.scriptDataDoubleEscapedDashDashStatec C sP | j j� }|dkr8| jjtd dd�� d| _| j| _n| j j|� | j | _dS )Nrz rR )r) r* r2 T) r r< r+ r= r r� � scriptDataDoubleEscapeEndStater rC r� )r# r* r& r& r' r� > s z6HTMLTokenizer.scriptDataDoubleEscapedLessThanSignStatec C s� | j j� }|ttd�B krR| jjtd |d�� | jj� dkrH| j | _ q�| j | _ nB|t kr�| jjtd |d�� | j|7 _n| j j |� | j | _ dS )Nrz r| rR )r) r* r� T)rz r| )r r< r r@ r+ r= r r� r� r� r r� r rC )r# r* r& r& r' r� I s z,HTMLTokenizer.scriptDataDoubleEscapeEndStatec C s0 | j j� }|tkr$| j jtd� �n|tkrJ| jd j|dg� | j| _n�|dkr\| j � n�|dkrn| j | _n�|dkr�| j jt d d d �� | jd j|dg� | j| _n�|d kr�| j jt d dd �� | jd jddg� | j| _nF|t k�r| j jt d dd �� | j| _n | jd j|dg� | j| _dS )NTr* r2 r| rz �'�"rP rL r( z#invalid-character-in-attribute-name)r) r* rl zinvalid-codepointu �z#expected-attribute-name-but-got-eof)r� r� rP rL )r r< r ro r r r= �attributeNameStater rk r� r+ r r r )r# r* r& r& r' r� Y s6 z&HTMLTokenizer.beforeAttributeNameStatec C s� | j j� }d}d}|dkr&| j| _�n0|tkr^| jd d d || j jtd� 7 < d}� n�|dkrld}n�|tkr~| j| _n�|dkr�| j | _n�|d kr�| j j t d d d �� | jd d d d 7 < d}n�|dk�r| j j t d dd �� | jd d d |7 < d}nH|t k�r8| j j t d dd �� | j| _n | jd d d |7 < d}|�r�| jd d d jt�| jd d d< xP| jd d d� D ]:\}}| jd d d |k�r�| j j t d dd �� P �q�W |�r�| j� dS ) NTFrP r* r r r| rz rl r( zinvalid-codepoint)r) r* u �r� r� rL z#invalid-character-in-attribute-namezeof-in-attribute-namezduplicate-attributerS rS )r� r� rL rS rS rS rS rS rS )r r< �beforeAttributeValueStater r r ro r �afterAttributeNameStater� r+ r= r r r rf r rk )r# r* �leavingThisState� emitTokenrb �_r& r& r' r� w sR & z HTMLTokenizer.attributeNameStatec C sF | j j� }|tkr$| j jtd� �n |dkr8| j| _�n |dkrJ| j� n�|tkrp| jd j |dg� | j | _n�|dkr�| j | _n�|dkr�| j j t d d d �� | jd j d dg� | j | _n�|dk� r�| j j t d dd �� | jd j |dg� | j | _nF|tk�r&| j j t d dd �� | j| _n | jd j |dg� | j | _dS )NTrP r| r* r2 rz rl r( zinvalid-codepoint)r) r* u �r� r� rL z&invalid-character-after-attribute-namez expected-end-of-tag-but-got-eof)r� r� rL )r r< r ro r� r rk r r r= r� r� r+ r r r )r# r* r& r& r' r� � s: z%HTMLTokenizer.afterAttributeNameStatec C sj | j j� }|tkr$| j jtd� �nB|dkr8| j| _�n.|dkrX| j| _| j j|� �n|dkrl| j| _� n�|dkr�| j j t d dd�� | j � n�|d kr�| j j t d d d�� | j d d d d 7 < | j| _n�|dk�r| j j t d dd�� | j d d d |7 < | j| _nL|tk�rD| j j t d dd�� | j| _n"| j d d d |7 < | j| _dS )NTr� rK r� r| r( z.expected-attribute-value-but-got-right-bracket)r) r* rl zinvalid-codepointr* r u �rP rL �`z"equals-in-unquoted-attribute-valuez$expected-attribute-value-but-got-eofrS )rP rL r� rS rS )r r< r ro � attributeValueDoubleQuotedStater �attributeValueUnQuotedStaterC � attributeValueSingleQuotedStater+ r= r rk r r r )r# r* r& r& r' r� � s> z'HTMLTokenizer.beforeAttributeValueStatec C s� | j j� }|dkr | j| _n�|dkr0| jd� n�|dkrj| jjtd dd�� | jd d d d 7 < nN|t kr�| jjtd d d�� | j | _n&| jd d d || j j d� 7 < d S )Nr� rK rl r( zinvalid-codepoint)r) r* r* r u �z#eof-in-attribute-value-double-quoteTrS rS )r� rK rl ) r r< �afterAttributeValueStater ra r+ r= r r r r ro )r# r* r& r& r' r� � s z-HTMLTokenizer.attributeValueDoubleQuotedStatec C s� | j j� }|dkr | j| _n�|dkr0| jd� n�|dkrj| jjtd dd�� | jd d d d 7 < nN|t kr�| jjtd d d�� | j | _n&| jd d d || j j d� 7 < d S )Nr� rK rl r( zinvalid-codepoint)r) r* r* r u �z#eof-in-attribute-value-single-quoteTrS rS )r� rK rl ) r r< r� r ra r+ r= r r r r ro )r# r* r& r& r' r� s z-HTMLTokenizer.attributeValueSingleQuotedStatec C s | j j� }|tkr | j| _� n�|dkr2| jd� n�|dkrD| j� n�|dkr~| jjt d d d �� | j d d d |7 < n�|d kr�| jjt d dd �� | j d d d d7 < nV|t kr�| jjt d dd �� | j | _n.| j d d d || j j td�tB � 7 < dS )NrK r| r� r� rP rL r� r( z0unexpected-character-in-unquoted-attribute-value)r) r* r* r rl zinvalid-codepointu �z eof-in-attribute-value-no-quotesT)r� r� rP rL r� rS rS rS )rK r| r� r� rP rL r� rl )r r< r r� r ra rk r+ r= r r r r ro r@ )r# r* r& r& r' r� s, z)HTMLTokenizer.attributeValueUnQuotedStatec C s� | j j� }|tkr | j| _n�|dkr.| j� np|dkr@| j| _n^|tkrt| jj t d dd�� | j j |� | j | _n*| jj t d dd�� | j j |� | j| _dS )Nr| rz r( z$unexpected-EOF-after-attribute-value)r) r* z*unexpected-character-after-attribute-valueT) r r< r r� r rk r� r r+ r= r rC r )r# r* r& r& r' r� . s" z&HTMLTokenizer.afterAttributeValueStatec C s� | j j� }|dkr&d| jd< | j� n^|tkrZ| jjtd dd�� | j j|� | j | _ n*| jjtd dd�� | j j|� | j | _ dS )Nr| Tre r( z#unexpected-EOF-after-solidus-in-tag)r) r* z)unexpected-character-after-solidus-in-tag) r r< r rk r r+ r= r rC r r r� )r# r* r& r& r' r� B s z&HTMLTokenizer.selfClosingStartTagStatec C sD | j jd�}|jdd�}| jjtd |d�� | j j� | j| _dS )Nr| rl u ��Comment)r) r* T) r ro �replacer+ r= r r< r r )r# r* r& r& r' r� T s z HTMLTokenizer.bogusCommentStatec C s� | j j� g}|d dkrT|j| j j� � |d dkrPtd dd�| _| j| _dS � n�|d d kr�d}x.d&D ]&}|j| j j� � |d' |krjd}P qjW |r�td dd d dd�| _| j| _dS n�|d( dk�rH| jd k �rH| jj j �rH| jj j d) j | jj j k�rHd}x2d*D ]*}|j| j j� � |d+ |k�rd}P �qW |�rH| j | _dS | jjtd dd�� x |�rz| j j|j� � �q`W | j| _dS ),Nr r� r� r2 )r) r* T�d�D�o�OrH �C�t�T�y�Y�p�P�e�EF�Doctype)r) rb �publicId�systemId�correct�[�Ar( zexpected-dashes-or-doctyperS rS rS )r� r� �r� r� �rH r� �r� r� �r� r� �r� r� �r� r� )r� r� r� r� r� r� rS rS rS )r� r� r� r� r� r� rS )r r< r= r r �commentStartStater � doctypeStater �tree� openElements� namespace�defaultNamespace�cdataSectionStater+ rC r- r� )r# rG �matched�expectedr& r& r' r~ c sR z(HTMLTokenizer.markupDeclarationOpenStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < n�|dkr�| jjtd d d�� | jj| j� | j| _nP|t kr�| jjtd d d�� | jj| j� | j| _n| jd |7 < | j | _d S ) Nr� rl r( zinvalid-codepoint)r) r* r* u �r| zincorrect-commentzeof-in-commentT) r r< �commentStartDashStater r+ r= r r r r � commentState)r# r* r& r& r' r� � s( z HTMLTokenizer.commentStartStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < n�|dkr�| jjtd d d�� | jj| j� | j| _nT|t kr�| jjtd d d�� | jj| j� | j| _n | jd d| 7 < | j | _d S ) Nr� rl r( zinvalid-codepoint)r) r* r* u -�r| zincorrect-commentzeof-in-commentT) r r< �commentEndStater r+ r= r r r r r� )r# r* r& r& r' r� � s( z#HTMLTokenizer.commentStartDashStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < nT|tkr�| jjtd dd�� | jj| j� | j | _n | jd || j j d � 7 < d S ) Nr� rl r( zinvalid-codepoint)r) r* r* u �zeof-in-commentT)r� rl ) r r< �commentEndDashStater r+ r= r r r r ro )r# r* r& r& r' r� � s zHTMLTokenizer.commentStatec C s� | j j� }|dkr | j| _n�|dkrV| jjtd dd�� | jd d7 < | j| _nT|t kr�| jjtd dd�� | jj| j� | j | _n | jd d| 7 < | j| _d S ) Nr� rl r( zinvalid-codepoint)r) r* r* u -�zeof-in-comment-end-dashT) r r< r� r r+ r= r r r� r r )r# r* r& r& r' r� � s z!HTMLTokenizer.commentEndDashStatec C s, | j j� }|dkr*| jj| j� | j| _n�|dkrd| jjtd dd�� | jd d7 < | j| _n�|dkr�| jjtd d d�� | j | _n�|d kr�| jjtd d d�� | jd |7 < nj|t kr�| jjtd d d�� | jj| j� | j| _n4| jjtd d d�� | jd d| 7 < | j| _dS )Nr| rl r( zinvalid-codepoint)r) r* r* u --�ry z,unexpected-bang-after-double-dash-in-commentr� z,unexpected-dash-after-double-dash-in-commentzeof-in-comment-double-dashzunexpected-char-in-commentz--T) r r< r+ r= r r r r r� �commentEndBangStater )r# r* r& r& r' r� � s6 z HTMLTokenizer.commentEndStatec C s� | j j� }|dkr*| jj| j� | j| _n�|dkrN| jd d7 < | j| _n�|dkr�| jjtd dd�� | jd d 7 < | j | _nT|t kr�| jjtd d d�� | jj| j� | j| _n | jd d| 7 < | j | _d S ) Nr| r� r* z--!rl r( zinvalid-codepoint)r) r* u --!�z eof-in-comment-end-bang-stateT) r r< r+ r= r r r r� r r� r )r# r* r& r& r' r� s( z!HTMLTokenizer.commentEndBangStatec C s� | j j� }|tkr | j| _nj|tkr\| jjtd dd�� d| j d< | jj| j � | j | _n*| jjtd dd�� | j j |� | j| _dS )Nr( z!expected-doctype-name-but-got-eof)r) r* Fr� zneed-space-after-doctypeT) r r< r �beforeDoctypeNameStater r r+ r= r r r rC )r# r* r& r& r' r� s zHTMLTokenizer.doctypeStatec C s� | j j� }|tkrn�|dkrT| jjtd dd�� d| jd< | jj| j� | j| _n�|dkr�| jjtd dd�� d | jd < | j | _nR|t kr�| jjtd d d�� d| jd< | jj| j� | j| _n|| jd < | j | _d S ) Nr| r( z+expected-doctype-name-but-got-right-bracket)r) r* Fr� rl zinvalid-codepointu �rb z!expected-doctype-name-but-got-eofT) r r< r r+ r= r r r r �doctypeNameStater )r# r* r& r& r' r� * s. z$HTMLTokenizer.beforeDoctypeNameStatec C s | j j� }|tkr2| jd jt�| jd< | j| _n�|dkrh| jd jt�| jd< | jj | j� | j | _n�|dkr�| jj t d dd�� | jd d7 < | j | _nh|t kr�| jj t d dd�� d | jd < | jd jt�| jd< | jj | j� | j | _n| jd |7 < d S ) Nrb r| rl r( zinvalid-codepoint)r) r* u �zeof-in-doctype-nameFr� T)r r< r r rf r �afterDoctypeNameStater r+ r= r r r� r )r# r* r& r& r' r� D s, z HTMLTokenizer.doctypeNameStatec C sR | j j� }|tkr�n8|dkr8| jj| j� | j| _�n|tkr�d| jd< | j j |� | jjt d dd�� | jj| j� | j| _� n�|d!kr�d }x$d'D ] }| j j� }||kr�d}P q�W |r�| j | _d S nJ|d(k�rd }x(d.D ] }| j j� }||k� r�d}P � q�W |�r| j | _d S | j j |� | jjt d d d |id �� d| jd< | j | _d S )/Nr| Fr� r( zeof-in-doctype)r) r* r� r� T�u�U�b�B�l�L�i�IrH r� �s�Sr� r� r� r� r� r� �m�Mz*expected-space-or-right-bracket-in-doctyper* )r) r* r4 )r� r� �r� r� �r� r� �r� r� �r� r� �rH r� )r� r� r� r� r� )r� r� �r� r� �r� r� �r� r� �r� r� �r� r� )r� r� r� r� r� )r r< r r+ r= r r r r rC r � afterDoctypePublicKeywordState� afterDoctypeSystemKeywordState�bogusDoctypeState)r# r* r� r� r& r& r' r� ] sT z#HTMLTokenizer.afterDoctypeNameStatec C s� | j j� }|tkr | j| _n�|d krP| jjtd dd�� | j j|� | j| _nT|t kr�| jjtd dd�� d| j d< | jj| j � | j | _n| j j|� | j| _d S ) Nr� r� r( zunexpected-char-in-doctype)r) r* zeof-in-doctypeFr� T)r� r� ) r r< r �"beforeDoctypePublicIdentifierStater r+ r= r rC r r r )r# r* r& r& r' r� � s" z,HTMLTokenizer.afterDoctypePublicKeywordStatec C s� | j j� }|tkrn�|dkr0d| jd< | j| _n�|dkrLd| jd< | j| _n�|dkr�| jjt d dd�� d | jd < | jj| j� | j | _nh|t kr�| jjt d d d�� d | jd < | jj| j� | j | _n(| jjt d d d�� d | jd < | j | _d S )Nr� r2 r� r� r| r( zunexpected-end-of-doctype)r) r* Fr� zeof-in-doctypezunexpected-char-in-doctypeT) r r< r r �(doctypePublicIdentifierDoubleQuotedStater �(doctypePublicIdentifierSingleQuotedStater+ r= r r r r� )r# r* r& r& r' r� � s4 z0HTMLTokenizer.beforeDoctypePublicIdentifierStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < n�|dkr�| jjtd d d�� d | jd < | jj| j� | j| _nR|t kr�| jjtd d d�� d | jd < | jj| j� | j| _n| jd |7 < d S )Nr� rl r( zinvalid-codepoint)r) r* r� u �r| zunexpected-end-of-doctypeFr� zeof-in-doctypeT) r r< �!afterDoctypePublicIdentifierStater r+ r= r r r r )r# r* r& r& r' r� � s* z6HTMLTokenizer.doctypePublicIdentifierDoubleQuotedStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < n�|dkr�| jjtd d d�� d | jd < | jj| j� | j| _nR|t kr�| jjtd d d�� d | jd < | jj| j� | j| _n| jd |7 < d S )Nr� rl r( zinvalid-codepoint)r) r* r� u �r| zunexpected-end-of-doctypeFr� zeof-in-doctypeT) r r< r� r r+ r= r r r r )r# r* r& r& r' r� � s* z6HTMLTokenizer.doctypePublicIdentifierSingleQuotedStatec C s | j j� }|tkr | j| _n�|dkr<| jj| j� | j| _n�|dkrn| jjt d dd�� d| jd< | j | _n�|dkr�| jjt d dd�� d| jd< | j | _nh|t kr�| jjt d d d�� d | jd < | jj| j� | j| _n(| jjt d dd�� d | jd < | j | _d S ) Nr| r� r( zunexpected-char-in-doctype)r) r* r2 r� r� zeof-in-doctypeFr� T)r r< r �-betweenDoctypePublicAndSystemIdentifiersStater r+ r= r r r �(doctypeSystemIdentifierDoubleQuotedState�(doctypeSystemIdentifierSingleQuotedStater r� )r# r* r& r& r' r� � s6 z/HTMLTokenizer.afterDoctypePublicIdentifierStatec C s� | j j� }|tkrn�|dkr4| jj| j� | j| _n�|dkrPd| jd< | j| _n�|dkrld| jd< | j | _nh|t kr�| jjt d dd�� d | jd < | jj| j� | j| _n(| jjt d d d�� d | jd < | j | _d S ) Nr| r� r2 r� r� r( zeof-in-doctype)r) r* Fr� zunexpected-char-in-doctypeT) r r< r r+ r= r r r r� r� r r r� )r# r* r& r& r' r� s. z;HTMLTokenizer.betweenDoctypePublicAndSystemIdentifiersStatec C s� | j j� }|tkr | j| _n�|d krP| jjtd dd�� | j j|� | j| _nT|t kr�| jjtd dd�� d| j d< | jj| j � | j | _n| j j|� | j| _d S ) Nr� r� r( zunexpected-char-in-doctype)r) r* zeof-in-doctypeFr� T)r� r� ) r r< r �"beforeDoctypeSystemIdentifierStater r+ r= r rC r r r )r# r* r& r& r' r� ) s" z,HTMLTokenizer.afterDoctypeSystemKeywordStatec C s� | j j� }|tkrn�|dkr0d| jd< | j| _n�|dkrLd| jd< | j| _n�|dkr�| jjt d dd�� d | jd < | jj| j� | j | _nh|t kr�| jjt d d d�� d | jd < | jj| j� | j | _n(| jjt d dd�� d | jd < | j | _d S ) Nr� r2 r� r� r| r( zunexpected-char-in-doctype)r) r* Fr� zeof-in-doctypeT) r r< r r r� r r� r+ r= r r r r� )r# r* r& r& r' r� = s4 z0HTMLTokenizer.beforeDoctypeSystemIdentifierStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < n�|dkr�| jjtd d d�� d | jd < | jj| j� | j| _nR|t kr�| jjtd d d�� d | jd < | jj| j� | j| _n| jd |7 < d S )Nr� rl r( zinvalid-codepoint)r) r* r� u �r| zunexpected-end-of-doctypeFr� zeof-in-doctypeT) r r< �!afterDoctypeSystemIdentifierStater r+ r= r r r r )r# r* r& r& r' r� Z s* z6HTMLTokenizer.doctypeSystemIdentifierDoubleQuotedStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < n�|dkr�| jjtd d d�� d | jd < | jj| j� | j| _nR|t kr�| jjtd d d�� d | jd < | jj| j� | j| _n| jd |7 < d S )Nr� rl r( zinvalid-codepoint)r) r* r� u �r| zunexpected-end-of-doctypeFr� zeof-in-doctypeT) r r< r� r r+ r= r r r r )r# r* r& r& r' r� r s* z6HTMLTokenizer.doctypeSystemIdentifierSingleQuotedStatec C s� | j j� }|tkrn~|dkr4| jj| j� | j| _n^|tkrt| jjt d dd�� d| jd< | jj| j� | j| _n | jjt d dd�� | j | _dS ) Nr| r( zeof-in-doctype)r) r* Fr� zunexpected-char-in-doctypeT) r r< r r+ r= r r r r r r� )r# r* r& r& r' r� � s z/HTMLTokenizer.afterDoctypeSystemIdentifierStatec C sZ | j j� }|dkr*| jj| j� | j| _n,|tkrV| j j|� | jj| j� | j| _n dS )Nr| T) r r< r+ r= r r r r rC )r# r* r& r& r' r� � s z HTMLTokenizer.bogusDoctypeStatec C s� g }x�|j | jjd�� |j | jjd�� | jj� }|tkr@P q|dksLt�|d dd � dkrx|d d d� |d< P q|j |� qW dj|�}|jd�}|dkr�x&t|�D ]}| j j t d d d �� q�W |j dd �}|r�| j j t d |d �� | j | _ dS )N�]r| r � z]]r2 rl r r( zinvalid-codepoint)r) r* u �rR TrS ���rS r� rS )r= r ro r< r �AssertionErrorr? �count�ranger+ r r� r r )r# r* r< � nullCountr� r& r& r' r� � s0 z HTMLTokenizer.cdataSectionState)N)NF)N�__name__� __module__� __qualname__�__doc__r" r/ rJ r` ra rk r rm rs rq ru rw rx rn r r� rr r� r� rt r� r� rv r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r~ r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� � __classcell__r& r& )r% r' r s� H P # 6 "- 3 r N)r r ) � __future__r r r Zpip._vendor.sixr rA � collectionsr r �sysr � constantsr r r r r r r r r r � _inputstreamr �_trier rT �dictrg �objectr r& r& r& r' �<module> s