Skip to content

Incompatible http streaming protocol with python requests #16

@Maxersh

Description

@Maxersh

How to reproduce the issue:

import requests

token = "token"
cluster_id = "cluster_id"
url = f"https://mdb.api.cloud.yandex.net/managed-postgresql/v1/clusters/{cluster_id}:stream_logs"

with requests.get(url,
                  params={"serviceType":"POSTGRESQL"},
                  headers={"Authorization":f"Bearer {token}"},
                  stream=True
                  ) as response:
    for chunk in response.iter_content(chunk_size=None, decode_unicode=True):
        print(chunk)

The above code outputs the logs as they come in. However, an exception like the one below occurs when a timeout occurs.

Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 748, in _update_chunk_length
self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 443, in _error_catcher
yield
File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 815, in read_chunked
self._update_chunk_length()
File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 752, in _update_chunk_length
raise InvalidChunkLength(self, line)
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/requests/models.py", line 760, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 623, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 803, in read_chunked
with self._error_catcher():
File "/usr/lib64/python3.10/contextlib.py", line 153, in exit
self.gen.throw(typ, value, traceback)
File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 460, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/user/scripts/test.py", line 12, in <module>
for ind, chunk in enumerate(response.iter_content(chunk_size=None, decode_unicode=True)):
File "/usr/lib/python3.10/site-packages/requests/utils.py", line 549, in stream_decode_response_unicode
for chunk in iterator:
File "/usr/lib/python3.10/site-packages/requests/models.py", line 763, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

Apparently this only happens if any logs were output. If there are no logs, one message about stream timeout will be output and the program will end properly.
I found a discussion of a similar error. Maintainer sad it's not a bug. This is a faulty implementation of the protocol on the server side.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions