관리-도구
편집 파일: collector.cpython-38.pyc
U &?�f�? � @ s� d Z ddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z ddlmZ ddlmZ ddlmZmZmZmZmZmZmZmZmZm Z m Z ddl m Z ddl m!Z! dd l"m#Z#m$Z$ dd l%m&Z& dd l'm(Z( dd l)m*Z* dd l+m,Z, ddl-m.Z. ddl/m0Z0 ddl1m2Z2 ddl3m4Z4 ddl5m6Z6m7Z7m8Z8 e�9e:�Z;ee<e<f Z=e<ee< d�dd�Z>G dd� de?�Z@e!dd�dd�ZAG d d � d e?�ZBe<e,dd �d d �ZCe<e,e!d �d!d"�ZDe=ee< d#�d$d%�ZEG d&d'� d'�ZFG d(d)� d)e�ZGeGeGd*�d+d,�ZHeHd-ee( d.�d/d0��ZIe d1d2�G d3d-� d-��ZJG d4d5� d5e�ZKdDe(e e<e?f eed6 dd7�d8d9�ZLdEe!eMeJd:�d;d<�ZNe(e,ed- d=�d>d?�ZOG d@dA� dAe�ZPG dBdC� dC�ZQdS )FzO The main purpose of this module is to expose LinkCollector.collect_sources(). � N)� dataclass)� HTMLParser)�Values) �Callable�Dict�Iterable�List�MutableMapping� NamedTuple�Optional�Protocol�Sequence�Tuple�Union)�requests)�Response)� RetryError�SSLError)�NetworkConnectionError)�Link)� SearchScope)� PipSession)�raise_for_status)�is_archive_file��redact_auth_from_url)�vcs� )�CandidatesFromPage� LinkSource� build_source��url�returnc C s6 t jD ]*}| �� �|�r| t|� dkr| S qdS )zgLook for VCS schemes in the URL. Returns the matched VCS scheme, or None if there's no match. z+:N)r �schemes�lower� startswith�len)r" �scheme� r) �K/opt/hc_python/lib/python3.8/site-packages/pip/_internal/index/collector.py�_match_vcs_scheme3 s r+ c s& e Zd Zeedd�� fdd� Z� ZS )�_NotAPIContentN)� content_type� request_descr# c s t � �||� || _|| _d S �N)�super�__init__r- r. )�selfr- r. �� __class__r) r* r1 ? s z_NotAPIContent.__init__)�__name__� __module__� __qualname__�strr1 � __classcell__r) r) r3 r* r, > s r, )�responser# c C s6 | j �dd�}|�� }|�d�r$dS t|| jj��dS )z� Check the Content-Type header to ensure the response contains a Simple API Response. Raises `_NotAPIContent` if the content type is not a valid content-type. � Content-Type�Unknown)z text/htmlz#application/vnd.pypi.simple.v1+html�#application/vnd.pypi.simple.v1+jsonN)�headers�getr% r& r, �request�method)r: r- �content_type_lr) r) r* �_ensure_api_headerE s �rC c @ s e Zd ZdS )�_NotHTTPN)r5 r6 r7 r) r) r) r* rD [ s rD )r" �sessionr# c C sF t j�| �\}}}}}|dkr$t� �|j| dd�}t|� t|� dS )z� Send a HEAD request to the URL, and ensure the response contains a simple API Response. Raises `_NotHTTP` if the URL is not available for a HEAD request, or `_NotAPIContent` if the content type is not a valid content type. > �https�httpT)�allow_redirectsN)�urllib�parse�urlsplitrD �headr rC )r" rE r( �netloc�path�query�fragment�respr) r) r* �_ensure_api_response_ s rR c C sz t t| �j�rt| |d� t�dt| �� |j| d�dddg�dd�d �}t |� t |� t�d t| �|j �d d �� |S ) aY Access an Simple API response with GET, and return the response. This consists of three parts: 1. If the URL looks suspiciously like an archive, send a HEAD first to check the Content-Type is HTML or Simple API, to avoid downloading a large file. Raise `_NotHTTP` if the content type cannot be determined, or `_NotAPIContent` if it is not HTML or a Simple API. 2. Actually perform the request. Raise HTTP exceptions on network failures. 3. Check the Content-Type header to make sure we got a Simple API response, and raise `_NotAPIContent` otherwise. �rE zGetting page %sz, r= z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z max-age=0)�Acceptz Cache-Control)r> zFetched page %s as %sr; r< ) r r �filenamerR �logger�debugr r? �joinr rC r> )r" rE rQ r) r) r* �_get_simple_responseq s, ���� �rY )r> r# c C s< | r8d| kr8t j�� }| d |d< |�d�}|r8t|�S dS )z=Determine if we have any encoding information in our headers.r; z content-type�charsetN)�email�message�Message� get_paramr8 )r> �mrZ r) r) r* �_get_encoding_from_headers� s r` c @ s: e Zd Zddd�dd�Zeed�dd�Zed �d d �ZdS ) �CacheablePageContent� IndexContentN��pager# c C s |j s t�|| _d S r/ )�cache_link_parsing�AssertionErrorrd �r2 rd r) r) r* r1 � s z CacheablePageContent.__init__)�otherr# c C s t |t| ��o | jj|jjkS r/ )� isinstance�typerd r" )r2 rh r) r) r* �__eq__� s zCacheablePageContent.__eq__�r# c C s t | jj�S r/ )�hashrd r" �r2 r) r) r* �__hash__� s z CacheablePageContent.__hash__) r5 r6 r7 r1 �object�boolrk �intro r) r) r) r* ra � s ra c @ s e Zd Zdee d�dd�ZdS )� ParseLinksrb rc c C s d S r/ r) rg r) r) r* �__call__� � zParseLinks.__call__N)r5 r6 r7 r r rt r) r) r) r* rs � s rs )�fnr# c sL t jdd�ttt d�� fdd� ��t �� �dtt d�� �fdd � �}|S ) z� Given a function that parses an Iterable[Link] from an IndexContent, cache the function's result (keyed by CacheablePageContent), unless the IndexContent `page` has `page.cache_link_parsing == False`. N)�maxsize)�cacheable_pager# c s t � | j��S r/ )�listrd )rx )rv r) r* �wrapper� s z*with_cached_index_content.<locals>.wrapperrb rc c s | j r�t| ��S t� | ��S r/ )re ra ry )rd �rv rz r) r* �wrapper_wrapper� s z2with_cached_index_content.<locals>.wrapper_wrapper)� functools� lru_cachera r r �wraps)rv r| r) r{ r* �with_cached_index_content� s r� rb rc c c s� | j �� }|�d�rTt�| j�}|�dg �D ]"}t�|| j �}|dkrHq,|V q,dS t | j �}| j pfd}|� | j� |�� | j }|jp�|}|jD ]$} tj| ||d�}|dkr�q�|V q�dS )z\ Parse a Simple API's Index Content, and yield its anchor elements as Link objects. r= �filesNzutf-8)�page_url�base_url)r- r% r& �json�loads�contentr? r � from_jsonr" �HTMLLinkParser�encoding�feed�decoder� �anchors� from_element) rd rB �data�file�link�parserr� r" r� �anchorr) r) r* � parse_links� s& r� T)�frozenc @ sP e Zd ZU dZeed< eed< ee ed< eed< dZe ed< ed�d d �Z d S ) rb a� Represents one response (or page), along with its URL. :param encoding: the encoding to decode the given content. :param url: the URL from which the HTML was downloaded. :param cache_link_parsing: whether links parsed from this page's url should be cached. PyPI index urls should have this set to False, for example. r� r- r� r" Tre rl c C s t | j�S r/ )r r" rn r) r) r* �__str__ s zIndexContent.__str__N) r5 r6 r7 �__doc__�bytes�__annotations__r8 r re rq r� r) r) r) r* rb � s c sn e Zd ZdZedd�� fdd� Zeeeeee f dd�dd�Z eeeee f ee d �d d �Z � Z S ) r� zf HTMLParser that keeps the first base HREF and a list of all anchor elements' attributes. Nr! c s$ t � jdd� || _d | _g | _d S )NT)�convert_charrefs)r0 r1 r" r� r� )r2 r" r3 r) r* r1 s zHTMLLinkParser.__init__)�tag�attrsr# c C sH |dkr,| j d kr,| �|�}|d k rD|| _ n|dkrD| j�t|�� d S )N�base�a)r� �get_hrefr� �append�dict)r2 r� r� �hrefr) r) r* �handle_starttag s z HTMLLinkParser.handle_starttag)r� r# c C s" |D ]\}}|dkr| S qd S )Nr� r) )r2 r� �name�valuer) r) r* r� & s zHTMLLinkParser.get_href) r5 r6 r7 r� r8 r1 r r r r� r� r9 r) r) r3 r* r� s "r� ).N)r� �reason�methr# c C s |d krt j}|d| |� d S )Nz%Could not fetch URL %s: %s - skipping)rV rW )r� r� r� r) r) r* �_handle_get_simple_fail- s r� )r: re r# c C s& t | j�}t| j| jd || j|d�S )Nr; )r� r" re )r` r> rb r� r" )r: re r� r) r) r* �_make_index_content7 s �r� )r� rE r# c C s� | j �dd�d }t|�}|r0t�d|| � d S tj�|�\}}}}}}|dkr�tj � tj � |��r�|� d�sv|d7 }tj�|d�}t�d|� zt||d �}W �nD tk r� t�d | � Y �n2 tk r� } zt�d | |j|j� W 5 d }~X Y n� tk �r$ } zt| |� W 5 d }~X Y n� tk �rP } zt| |� W 5 d }~X Y n� tk �r� } z$d } | t|�7 } t| | tjd � W 5 d }~X Y nd tj k �r� } zt| d|� �� W 5 d }~X Y n0 tj k �r� t| d� Y nX t || j d�S d S )N�#r r zICannot look at %s URL %s because it does not support lookup as web pages.r� �/z index.htmlz# file: URL is directory, getting %srS z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.z�Skipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )r� zconnection error: z timed out)re ) r" �splitr+ rV �warningrI rJ �urlparse�osrN �isdirr@ � url2pathname�endswith�urljoinrW rY rD r, r. r- r r� r r r8 �infor �ConnectionError�Timeoutr� re ) r� rE r" � vcs_schemer( �_rN rQ �excr� r) r) r* �_get_index_contentD sV � � � r� c @ s. e Zd ZU eee ed< eee ed< dS )�CollectedSources� find_links� index_urlsN)r5 r6 r7 r r r r� r) r) r) r* r� � s r� c @ sx e Zd ZdZeedd�dd�Zedeee d d�dd ��Z e e e d �d d ��Zeee d �dd�Ze eed�dd�ZdS )� LinkCollectorz� Responsible for collecting Link objects from all configured locations, making network requests as needed. The class's main method is its collect_sources() method. N)rE � search_scoper# c C s || _ || _d S r/ )r� rE )r2 rE r� r) r) r* r1 � s zLinkCollector.__init__F)rE �options�suppress_no_indexr# c C sd |j g|j }|jr8|s8t�dd�dd� |D ��� g }|jp@g }tj|||jd�}t ||d�}|S )z� :param session: The Session to use to make requests. :param suppress_no_index: Whether to ignore the --no-index option when constructing the SearchScope object. zIgnoring indexes: %s�,c s s | ]}t |�V qd S r/ r )�.0r" r) r) r* � <genexpr>� s z'LinkCollector.create.<locals>.<genexpr>)r� r� �no_index)rE r� ) � index_url�extra_index_urlsr� rV rW rX r� r �creater� )�clsrE r� r� r� r� r� �link_collectorr) r) r* r� � s$ � ��zLinkCollector.createrl c C s | j jS r/ )r� r� rn r) r) r* r� � s zLinkCollector.find_links)�locationr# c C s t || jd�S )z> Fetch an HTML page containing package links. rS )r� rE )r2 r� r) r) r* �fetch_response� s z LinkCollector.fetch_response)� project_name�candidates_from_pager# c s� t �� ��fdd��j���D ���� }t �� ��fdd��jD ���� }t�tj �r�dd� t � ||�D �}t |�� d�� d�g| }t� d�|�� tt|�t|�d �S ) Nc 3 s& | ] }t |� �jjd d �d�V qdS )F�r� �page_validator� expand_dirre r� N�r rE �is_secure_origin�r� �loc�r� r� r2 r) r* r� � s ��z0LinkCollector.collect_sources.<locals>.<genexpr>c 3 s& | ] }t |� �jjd d �d�V qdS )Tr� Nr� r� r� r) r* r� � s ��c S s* g | ]"}|d k r|j d k rd|j � ��qS )Nz* )r� )r� �sr) r) r* � <listcomp>� s �z1LinkCollector.collect_sources.<locals>.<listcomp>z' location(s) to search for versions of �:� )r� r� )� collections� OrderedDictr� �get_index_urls_locations�valuesr� rV � isEnabledFor�logging�DEBUG� itertools�chainr' rW rX r� ry )r2 r� r� �index_url_sources�find_links_sources�linesr) r� r* �collect_sources� s& � � ����z LinkCollector.collect_sources)F)r5 r6 r7 r� r r r1 � classmethodr rq r� �propertyr r8 r� r r rb r� r r� r� r) r) r) r* r� � s( � ��!�r� )N)T)Rr� r� � email.messager[ r} r� r� r� r� � urllib.parserI �urllib.request� dataclassesr � html.parserr �optparser �typingr r r r r r r r r r r � pip._vendorr Zpip._vendor.requestsr Z pip._vendor.requests.exceptionsr r �pip._internal.exceptionsr �pip._internal.models.linkr �!pip._internal.models.search_scoper � pip._internal.network.sessionr �pip._internal.network.utilsr � pip._internal.utils.filetypesr �pip._internal.utils.miscr �pip._internal.vcsr �sourcesr r r � getLoggerr5 rV r8 �ResponseHeadersr+ � Exceptionr, rC rD rR rY r` ra rs r� r� rb r� r� rq r� r� r� r� r) r) r) r* �<module> st 4 ? � � � � =