관리-도구
편집 파일: _inputstream.cpython-38.opt-1.pyc
U ��.e( � ! @ s� d dl mZmZmZ d dlmZmZ d dlmZm Z d dl Z d dl Z d dl m Z ddlmZmZmZmZ ddlmZ dd lmZ d d lmZ zd d lmZ W n ek r� eZY nX ed d � eD ��Zedd � eD ��Z edd � eD ��Z eeddg�B Z dZ ej �r(e �!e dd� e"d� d �Z#n e �!e �Z#e$ddddddd d d d d d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5g �Z%e �!d6�Z&i Z'G d7d8� d8e(�Z)d9d:� Z*G d;d<� d<e(�Z+G d=d>� d>e+�Z,G d?d@� d@e-�Z.G dAdB� dBe(�Z/G dCdD� dDe(�Z0dEdF� Z1dS )G� )�absolute_import�division�unicode_literals)� text_type� binary_type)� http_client�urllibN)� webencodings� )�EOF�spaceCharacters� asciiLetters�asciiUppercase)�_ReparseException)�_utils)�StringIO)�BytesIOc C s g | ]}|� d ��qS ��ascii��encode��.0�item� r �E/usr/lib/python3.8/site-packages/pip/_vendor/html5lib/_inputstream.py� <listcomp> s r c C s g | ]}|� d ��qS r r r r r r r s c C s g | ]}|� d ��qS r r r r r r r s � >� <u� [- - --]���z"\uD800-\uDFFF"�]i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� z[ - -/:-@\[-`{-~]c @ sH e Zd ZdZdd� Zdd� Zdd� Zdd � Zd d � Zd d � Z dd� Z dS )�BufferedStreamz�Buffering for streams that do not have buffering of their own The buffer is implemented as a list of chunks on the assumption that joining many strings will be slow since it is O(n**2) c C s || _ g | _ddg| _d S )Nr r )�stream�buffer�position)�selfr" r r r �__init__@ s zBufferedStream.__init__c C s< d}| j d | jd � D ]}|t|�7 }q|| jd 7 }|S �Nr r )r# r$ �len)r% �pos�chunkr r r �tellE s zBufferedStream.tellc C sD |}d}t | j| �|k r6|t | j| �8 }|d7 }q||g| _d S r' )r( r# r$ )r% r) �offset�ir r r �seekL s zBufferedStream.seekc C sT | j s| �|�S | jd t| j �krF| jd t| j d �krF| �|�S | �|�S d S )Nr r r )r# � _readStreamr$ r( �_readFromBuffer)r% �bytesr r r �readU s � zBufferedStream.readc C s t dd� | jD ��S )Nc S s g | ] }t |��qS r )r( r r r r r _ s z1BufferedStream._bufferedBytes.<locals>.<listcomp>)�sumr# �r% r r r �_bufferedBytes^ s z BufferedStream._bufferedBytesc C s< | j �|�}| j�|� | jd d7 < t|�| jd< |S r' )r" r2 r# �appendr$ r( )r% r1 �datar r r r/ a s zBufferedStream._readStreamc C s� |}g }| j d }| j d }|t| j�k r�|dkr�| j| }|t|�| kr`|}||| g| _ n"t|�| }|t|�g| _ |d7 }|�|||| � � ||8 }d}q |r�|�| �|�� d�|�S )Nr r � )r$ r( r# r6 r/ �join)r% r1 ZremainingBytes�rvZ bufferIndexZ bufferOffsetZ bufferedDataZ bytesToReadr r r r0 h s$ z BufferedStream._readFromBufferN) �__name__� __module__� __qualname__�__doc__r& r+ r. r2 r5 r/ r0 r r r r r! 9 s r! c K s� t | tj�s(t | tjj�r.t | jtj�r.d}n&t| d�rJt | �d�t �}n t | t �}|r�dd� |D �}|rvt d| ��t | f|�S t | f|�S d S )NFr2 r c S s g | ]}|� d �r|�qS )Z _encoding)�endswith)r �xr r r r � s z#HTMLInputStream.<locals>.<listcomp>z3Cannot set an encoding with a unicode input, set %r) � isinstancer Z HTTPResponser ZresponseZaddbase�fp�hasattrr2 r � TypeError�HTMLUnicodeInputStream�HTMLBinaryInputStream)�source�kwargsZ isUnicodeZ encodingsr r r �HTMLInputStream� s � � rI c @ sp e Zd ZdZdZdd� Zdd� Zdd� Zd d � Zd d � Z d d� Z ddd�Z dd� Z dd� Z d dd�Zdd� ZdS ) rE ��Provides a unicode stream of characters to the HTMLTokenizer. This class takes care of character encoding and removing or replacing incorrect byte-sequences and also provides column and line tracking. i ( c C sZ t jsd| _n td�dkr$| j| _n| j| _dg| _td�df| _| � |�| _ | � � dS )� Initialises the HTMLInputStream. HTMLInputStream(source, [encoding]) -> Normalized stream from source for use by html5lib. source can be either a file-object, local filename or a string. The optional encoding parameter must be a string that indicates the encoding. If specified, that encoding will be used, regardless of any BOM or later declaration (such as in a meta element) Nu r r �utf-8�certain) r �supports_lone_surrogates�reportCharacterErrorsr( �characterErrorsUCS4�characterErrorsUCS2ZnewLines�lookupEncoding� charEncoding� openStream� dataStream�reset)r% rG r r r r& � s z HTMLUnicodeInputStream.__init__c C s. d| _ d| _d| _g | _d| _d| _d | _d S )N� r )r* � chunkSize� chunkOffset�errors� prevNumLines� prevNumCols�_bufferedCharacterr4 r r r rV � s z HTMLUnicodeInputStream.resetc C s t |d�r|}nt|�}|S �zvProduces a file object from source. source can be either a file object, local filename or a string. r2 )rC r �r% rG r" r r r rT � s z!HTMLUnicodeInputStream.openStreamc C sT | j }|�dd|�}| j| }|�dd|�}|dkr@| j| }n ||d }||fS )N� r r r )r* �countr[ �rfindr\ )r% r, r* ZnLinesZ positionLineZ lastLinePosZpositionColumnr r r � _position� s z HTMLUnicodeInputStream._positionc C s | � | j�\}}|d |fS )z:Returns (line, col) of the current position in the stream.r )rc rY )r% �line�colr r r r$ � s z HTMLUnicodeInputStream.positionc C s6 | j | jkr| �� stS | j }| j| }|d | _ |S )zo Read one character from the stream or queue if available. Return EOF when EOF is reached. r )rY rX � readChunkr r* )r% rY �charr r r rg � s zHTMLUnicodeInputStream.charNc C s� |d kr| j }| �| j�\| _| _d| _d| _d| _| j�|�}| j rX| j | }d | _ n|s`dS t |�dkr�t |d �}|dks�d| kr�dkr�n n|d | _ |d d� }| j r�| � |� |� d d �}|� d d �}|| _t |�| _d S ) NrW r Fr r � � � i�� z r` � T)�_defaultChunkSizerc rX r[ r\ r* rY rU r2 r] r( �ordrO �replace)r% rX r7 Zlastvr r r rf � s0 z HTMLUnicodeInputStream.readChunkc C s( t tt�|���D ]}| j�d� qd S )N�invalid-codepoint)�ranger( �invalid_unicode_re�findallrZ r6 )r% r7 �_r r r rP % s z*HTMLUnicodeInputStream.characterErrorsUCS4c C s� d}t �|�D ]�}|rqt|�� �}|�� }t�|||d � �rrt�|||d � �}|tkrl| j � d� d}q|dkr�|dkr�|t |�d kr�| j � d� qd}| j � d� qd S )NF� rn Tri i�� r ) rp �finditerrl �group�startr ZisSurrogatePairZsurrogatePairToCodepoint�non_bmp_invalid_codepointsrZ r6 r( )r% r7 �skip�matchZ codepointr) Zchar_valr r r rQ ) s" �z*HTMLUnicodeInputStream.characterErrorsUCS2Fc C s� zt ||f }W nN tk r^ d�dd� |D ��}|s@d| }t�d| � }t ||f< Y nX g }|�| j| j�}|dkr�| j| jkr�q�n0|� � }|| jkr�|� | j| j|� � || _q�|� | j| jd� � | � � sdq�qdd�|�}|S )z� Returns a string of characters from the stream up to but not including any character in 'characters' or EOF. 'characters' must be a container that supports the 'in' method and iteration over its characters. rW c S s g | ]}d t |� �qS )z\x%02x)rl )r �cr r r r N s z5HTMLUnicodeInputStream.charsUntil.<locals>.<listcomp>z^%sz[%s]+N) �charsUntilRegEx�KeyErrorr9 �re�compilery r* rY rX �endr6 rf ) r% Z charactersZopposite�charsZregexr: �mr �rr r r � charsUntil@ s, z!HTMLUnicodeInputStream.charsUntilc C s@ |d k r<| j dkr.|| j | _| jd7 _n| j d8 _ d S r' )rY r* rX )r% rg r r r �ungeto s z HTMLUnicodeInputStream.unget)N)F)r; r< r= r> rk r& rV rT rc r$ rg rf rP rQ r� r� r r r r rE � s & /rE c @ sL e Zd ZdZddd�Zdd� Zd d � Zdd d �Zd d� Zdd� Z dd� Z dS )rF rJ N� windows-1252Tc C s\ | � |�| _t�| | j� d| _d| _|| _|| _|| _|| _ || _ | � |�| _ | � � dS )rK i �d N)rT � rawStreamrE r& � numBytesMeta�numBytesChardet�override_encoding�transport_encoding�same_origin_parent_encoding�likely_encoding�default_encoding�determineEncodingrS rV )r% rG r� r� r� r� r� Z useChardetr r r r& � s z HTMLBinaryInputStream.__init__c C s& | j d j�| jd�| _t�| � d S )Nr rm )rS Z codec_info� streamreaderr� rU rE rV r4 r r r rV � s zHTMLBinaryInputStream.resetc C sD t |d�r|}nt|�}z|�|�� � W n t|�}Y nX |S r^ )rC r r. r+ r! r_ r r r rT � s z HTMLBinaryInputStream.openStreamc C s� | � � df}|d d k r |S t| j�df}|d d k r:|S t| j�df}|d d k rX|S | �� df}|d d k rt|S t| j�df}|d d k r�|d j�d�s�|S t| j�df}|d d k r�|S |�r`zddl m } W n t k r� Y nvX g }|� }|j �s,| j �| j�}|�s�q,|�|� |�|� q�|�� t|jd �}| j �d� |d k �r`|dfS t| j�df}|d d k �r�|S td�dfS )NrM r Z tentativezutf-16)�UniversalDetector�encodingr� )� detectBOMrR r� r� �detectEncodingMetar� �name� startswithr� Z%pip._vendor.chardet.universaldetectorr� � ImportErrorZdoner� r2 r� r6 Zfeed�close�resultr. r� )r% ZchardetrS r� ZbuffersZdetectorr# r� r r r r� � sP z'HTMLBinaryInputStream.determineEncodingc C s� t |�}|d krd S |jdkr(t d�}nT|| jd krH| jd df| _n4| j�d� |df| _| �� td| jd |f ��d S )N��utf-16be�utf-16lerL r rM z Encoding changed from %s to %s)rR r� rS r� r. rV r )r% Z newEncodingr r r �changeEncoding s z$HTMLBinaryInputStream.changeEncodingc C s� t jdt jdt jdt jdt jdi}| j�d�}|�|dd� �}d}|sp|�|�}d}|sp|�|dd � �}d }|r�| j� |� t |�S | j� d � dS dS ) z�Attempts to detect at BOM at the start of the stream. If an encoding can be determined from the BOM return the name of the encoding otherwise return NonerL r� r� zutf-32lezutf-32be� N� rs r ) �codecs�BOM_UTF8� BOM_UTF16_LE� BOM_UTF16_BE� BOM_UTF32_LE� BOM_UTF32_BEr� r2 �getr. rR )r% ZbomDict�stringr� r. r r r r� s2 � z HTMLBinaryInputStream.detectBOMc C sH | j �| j�}t|�}| j �d� |�� }|dk rD|jdkrDtd�}|S )z9Report the encoding declared by the meta element r Nr� rL )r� r2 r� �EncodingParserr. � getEncodingr� rR )r% r# �parserr� r r r r� 9 s z(HTMLBinaryInputStream.detectEncodingMeta)NNNNr� T)T) r; r< r= r> r& rV rT r� r� r� r� r r r r rF � s � * >"rF c @ s� e Zd ZdZdd� Zdd� Zdd� Zdd � Zd d � Zd d � Z dd� Z dd� Z e e e �Z dd� Ze e�Zefdd�Zdd� Zdd� Zdd� Zd S ) � EncodingBytesz�String-like object with an associated position and various extra methods If the position is ever greater than the string length then an exception is raisedc C s t �| |�� �S �N)r1 �__new__�lower�r% �valuer r r r� L s zEncodingBytes.__new__c C s d| _ d S )Nr )rc r� r r r r& P s zEncodingBytes.__init__c C s | S r� r r4 r r r �__iter__T s zEncodingBytes.__iter__c C s> | j d }| _ |t| �kr"t�n |dk r.t�| ||d � S )Nr r �rc r( � StopIterationrD �r% �pr r r �__next__W s zEncodingBytes.__next__c C s | � � S r� )r� r4 r r r �next_ s zEncodingBytes.nextc C sB | j }|t| �krt�n |dk r$t�|d | _ }| ||d � S r' r� r� r r r �previousc s zEncodingBytes.previousc C s | j t| �krt�|| _ d S r� �rc r( r� )r% r$ r r r � setPositionl s zEncodingBytes.setPositionc C s* | j t| �krt�| j dkr"| j S d S d S )Nr r� r4 r r r � getPositionq s zEncodingBytes.getPositionc C s | | j | j d � S �Nr )r$ r4 r r r �getCurrentByte{ s z EncodingBytes.getCurrentBytec C sH | j }|t| �k r>| ||d � }||kr4|| _|S |d7 }q|| _dS )z Skip past a list of charactersr N�r$ r( rc �r% r� r� rz r r r rx � s zEncodingBytes.skipc C sH | j }|t| �k r>| ||d � }||kr4|| _|S |d7 }q|| _d S r� r� r� r r r � skipUntil� s zEncodingBytes.skipUntilc C s> | j }| ||t|� � }|�|�}|r:| j t|�7 _ |S )z�Look for a sequence of bytes at the start of a string. If the bytes are found return True and advance the position to the byte after the match. Otherwise return False and leave the position alone)r$ r( r� )r% r1 r� r7 r: r r r � matchBytes� s zEncodingBytes.matchBytesc C sR | | j d� �|�}|dkrJ| jdkr,d| _| j|t|� d 7 _dS t�dS )z�Look for the next sequence of bytes matching a given sequence. If a match is found advance the position to the last byte of the matchNr r r T)r$ �findrc r( r� )r% r1 Z newPositionr r r �jumpTo� s zEncodingBytes.jumpToN)r; r< r= r> r� r& r� r� r� r� r� r� �propertyr$ r� � currentByte�spaceCharactersBytesrx r� r� r� r r r r r� H s r� c @ sX e Zd ZdZdd� Zdd� Zdd� Zdd � Zd d � Zd d � Z dd� Z dd� Z dd� Z dS )r� z?Mini parser for detecting character encoding from meta elementsc C s t |�| _d| _dS )z3string - the data to work on for encoding detectionN)r� r7 r� �r% r7 r r r r& � s zEncodingParser.__init__c C s� d| j fd| jfd| jfd| jfd| jfd| jff}| jD ]Z}d}|D ]D\}}| j�|�rFz|� }W q�W qF tk r� d}Y q�Y qFX qF|s: q�q:| jS ) Ns <!--s <metas </s <!s <?r TF) � handleComment� handleMeta�handlePossibleEndTag� handleOther�handlePossibleStartTagr7 r� r� r� )r% ZmethodDispatchrr Z keepParsing�key�methodr r r r� � s( � zEncodingParser.getEncodingc C s | j �d�S )zSkip over commentss -->�r7 r� r4 r r r r� � s z EncodingParser.handleCommentc C s� | j jtkrdS d}d }| �� }|d kr,dS |d dkr\|d dk}|r�|d k r�|| _dS q|d dkr�|d }t|�}|d k r�|| _dS q|d dkrtt|d ��}|�� }|d k rt|�}|d k r|r�|| _dS |}qd S ) NTFr s http-equivr s content-type� charsets content) r7 r� r� � getAttributer� rR �ContentAttrParserr� �parse)r% Z hasPragmaZpendingEncoding�attrZtentativeEncoding�codecZ contentParserr r r r� � s8 zEncodingParser.handleMetac C s | � d�S )NF)�handlePossibleTagr4 r r r r� � s z%EncodingParser.handlePossibleStartTagc C s t | j� | �d�S )NT)r� r7 r� r4 r r r r� � s z#EncodingParser.handlePossibleEndTagc C sb | j }|jtkr(|r$|�� | �� dS |�t�}|dkrD|�� n| �� }|d k r^| �� }qLdS )NTr )r7 r� �asciiLettersBytesr� r� r� �spacesAngleBracketsr� )r% ZendTagr7 rz r� r r r r� � s z EncodingParser.handlePossibleTagc C s | j �d�S )Nr r� r4 r r r r� s zEncodingParser.handleOtherc C s� | j }|�ttdg�B �}|dkr&dS g }g }|dkr>|r>q�nX|tkrR|�� }q�nD|dkrhd�|�dfS |tkr�|�|�� � n|dkr�dS |�|� t|�}q.|dkr�|� � d�|�dfS t|� |�� }|dk�r2|}t|�}||k�r t|� d�|�d�|�fS |tk�r$|�|�� � q�|�|� q�nJ|dk�rJd�|�dfS |tk�rd|�|�� � n|dk�rrdS |�|� t|�}|t k�r�d�|�d�|�fS |tk�r�|�|�� � n|dk�r�dS |�|� �q|dS ) z_Return a name,value pair for the next attribute in the stream, if one is found, or None� /)r NN� =)r� r r8 )� '� "r ) r7 rx r� � frozensetr9 �asciiUppercaseBytesr6 r� r� r� r� )r% r7 rz ZattrNameZ attrValueZ quoteCharr r r r� s` zEncodingParser.getAttributeN) r; r< r= r> r& r� r� r� r� r� r� r� r� r r r r r� � s $r� c @ s e Zd Zdd� Zdd� ZdS )r� c C s || _ d S r� )r7 r� r r r r& f s zContentAttrParser.__init__c C s � z�| j �d� | j jd7 _| j �� | j jdks<W d S | j jd7 _| j �� | j jdkr�| j j}| j jd7 _| j j}| j �|�r�| j || j j� W S W d S nR| j j}z"| j �t� | j || j j� W W S tk � r� | j |d � Y W S X W n tk �r Y d S X d S )Nr� r r� )r� r� )r7 r� r$ rx r� r� r� r� )r% Z quoteMarkZ oldPositionr r r r� j s. zContentAttrParser.parseN)r; r< r= r& r� r r r r r� e s r� c C sf t | t�r0z| �d�} W n tk r. Y dS X | dk r^z t�| �W S tk rZ Y dS X ndS dS )z{Return the python codec name corresponding to an encoding or None if the string doesn't correspond to a valid encoding.r N)rA r �decode�UnicodeDecodeErrorr �lookup�AttributeError)r� r r r rR � s rR )2Z __future__r r r Zpip._vendor.sixr r Zpip._vendor.six.movesr r r� r} Z pip._vendorr Z constantsr r r r r rW r �ior r r� r� r� r� r� r� Z invalid_unicode_no_surrogaterN r~ �evalrp �setrw Zascii_punctuation_rer{ �objectr! rI rE rF r1 r� r� r� rR r r r r �<module> s� �� � J g Ih 6'